Theory and Implementation of Linear Regression
By: Aryan Shrivastava
Tags: Cost Function, Linear Regression, Gradient Descent, Machine Learning, Scikit Learn

In this article, we will decode linear regression. This article will contain the following sections:-

1) What is a Linear Regression?

2) Sample example

3) Gradient Descent

4) Boston Housing Price example

5) Conclusions

1) What is a Linear Regression?

It is a form of supervised machine learning in which the underlying relationship between the features is linear.

Try to guess the example of linear regression:-

The correct answer is ©.

Now let’s take an arbitrary example of a dataset containing two features x1 and x2.

In this case, the underlying relationship between the features x1 and x2 is linear.

Now our task is to fit a line in this dataset that can generalize our dataset well and give us accurate results when treated with the unseen features x1 and x2.

This seems like a good fit for the datapoints.

In linear regression, we represent our model in the form of a 1st-degree linear equation.

h(x)=w0+w1*x1+w2*x2

No of weights (w) is defined by no of features +1(for w0 or bias).

Try to visualize it this way, say there are people (equals to no. of features) and each person says something and their collaborative say is what that comes out only.

By adjusting the settings of w1 and w2 there is a voice that comes out which is perfect.

Try to fit the same analogy into our machine learning algorithm i.e. there is the little contribution of feature x1 and little from x2. If we add them simply i.e. x1+x2=y whomsoever is more in value will contribute more towards the value of y but this will be little faulty( i.e whomsoever is more important should get the relevant value of weights). Consider the price of a house depends on two factors (features) no. of rooms and the area of land say 1275+2=$1277. Now consider tripling the no. of rooms 1275+6=1281. We have tripled the no. of rooms but the price has increases negligibility. (Back to person's example) so, in order to deal with this, we paste weights (w) that will empower the speech of every feature in deciding the final answer according to their relevance, not their value.

For w0 imagine this as a base price of the land in housing price.

Now let’s define stepwise procedure (or pseudo code) for linear regression.

Given,

Add a column of ones to adjust w0.

Guess weights w0, w1, w2

Take dot product of w and x

h(x) = xͼw (n*3 ͼ 3*1=n*1)

Steps: —

Calculate the difference between our answers from the guessed values of weights and actual answers.

2. Convey this difference to an optimizing algorithm (Gradient Descent) and improve our guess.

3. Update the values of weights.

4. Repeat till we get the difference close to zero or minimum (global or local).

Gradient Descent and the Cost function:-

Cost function or loss function — In the algorithm defined above, we do not take direct difference that gets passed to the gradient descent. We calculate this difference using loss/cost function like Mean Absolute errors, Mean squared error, etc. these cost function tells what is the cost that will need to be paid if the particular predicted value is taken into account.

Consider a simple example, say we have developed an ML model that predicts whether a person has cancer or not. If our model predicts that there is a 90% chance that a person has cancer and it turns out that actually the person has only a 10% chance of being cancer positive. Hence in such a case due to the wrong prediction of our Machine the cost that the person had to pay due to the wrong prediction is so high(80%).

Note that the above explanation has only a theoretical background, just to give you an insight that what a cost function accounts for, mathematically cost function is stated below.

Let y_pred(or h(w)) be the predicted value and y_actual be the real labels.

Then,

Mean squared error:-

Now we know that what cost function is, let’s define Gradient Descent.

Gradient Descent: — Gradient Descent is an optimizing algorithm that can be used to reach the minimum of a cost function.

What that means is that when gradient descent is being passed with the cost function value, it accordingly updates the weights so that you reach the minimum value of cost function after several iterations.

Suppose you are standing at a hill and want to go down as soon as possible. You decide to take a baby step (defined by alpha) towards the steepest direction at the point you are standing. What gradient descent does it that it tells that direction of steepest.

Contours

Say we are at A and we have to reach O, gradient descent helps to reach there.

To get a better sense take an example of a data set that has just one weight to be minimized.

Say, we have started from point A calculating gradient, and updating weights lets us reach A1, A2, A3, and finally Ai.

Now for multiple features imagine multiple weights as multiple dimensions and gradient descent lets us reach the minimum.

Till now we have not seen the mathematical face of Gradient Descent.

Hence, mathematically,

α is called the learning rate, it is one of the hyperparameters that we give to the ML model.

Clearer picture, for n features we have our hypothesis: —

h(w)=w0x0+w1x1+w2x2+………..+wnxn

Hence, cost function becomes (MSE):-

Hence,

Therefore, the update looks like:-

Boston Housing Price Example: —

Now, In this section of the article, we are going to discuss the programming implementation of linear regression. One thing to note that we are going to use a sckit-learn library to implement linear regression. The dataset we have taken is the Boston housing price dataset where the target variable is the price of the house. So let’s get started.

Note:- I will discuss snippets of the program.

This section of the code loads all the required libraries that we need in the program.

1. Numpy is used to frame our dataset into NumPy arrays that we use to input our data into linear regression.

2. Seaborn is used in our program to plot the pairplot that gives the plot of every feature with respect to every feature.

3. Pandas is used to convert our dataset into dataframe for a better presentation of the dataset.

4. Sklearn’s dataset is imported to import a dataset.

5. Train_test_split is used to split out the dataset into a test and training data set.

6. Standard Scaler is used to scale each feature of our dataset.

7. LinearRegression is used to implement the linear regression algorithm.

1. dataset.load_boston() loads the data of Boston housing price in a variable named data.

2.pd.DataFrame(data.data,columns=data.feature_names) creates data frame.

3. 3rd line creates another column in the dataset and assigns the value of y in each cell.

4. sb.pairplot plots the pair plot against each possible feature.s

1. 1st line splits the dataset into training and testing dataset. The size of the test size is 20 percent of the dataset.

2. This line initiates an object of class StandardScaler class so that it can be used to scale out the dataset on the same scale.

3. This line scales X_train.

4. This scales X_test.

5. 5th and 6th line convert the Y_test and Y_train into NumPy array from the data frame.

This section prints the number of items in each set (training and testing).

1. This line instantiates the object of class linear regression. All our explanations of matrix multiplications etc. are being self-implemented here.

2. Lr.fit(X_train, Y_train) this line fits the training data into a linear regression model. This is called training.

1. This line predicts the target values of the training dataset using the learned parameters from training data itself.

2. This line predicts the target values of the testing dataset using the same learned dataset.

1. This line loads mean squared error cost function.

2. This line compares how good our predictions are in the training dataset.

3. This line tests the prediction on the testing dataset that we put out separately only for testing purposes.

This line gives out the values of learned parameters

Conclusion:-

In this article, we have discussed probably the most basic algorithm of machine learning. In the coming series of articles, we will be dealing with some complex algorithms of machine learning. Also, note that we have not implemented linear regression using purely numpy this is because in real-life applications we usually never implement algorithms using pure mathematics. One way or another we use libraries that do the work for us.

Hopefully, you liked the article, your valuable feedbacks are deeply appreciated.

Thank You

Aryan Shrivastava