Multivariate Linear Regression in Python Step by Step | by Rashida Nasrin Sucky | Towards Data Science

Linear regression is probably the most simple machine learning algorithm. It is very good for starters because it uses simple formulas. So, it is good for learning machine-learning concepts. In this article, I will try to explain the multivariate linear regression step by step.
Linear regression uses the simple formula that we all learned in school:
Y = C + AX
Just as a reminder, Y is the output or dependent variable, X is the input or the independent variable, A is the slope, and C is the intercept.
For the linear regression, we follow these notations for the same formula:
If we have multiple independent variables, the formula for linear regression will look like:
Here, ‘h’ is called the hypothesis. This is the predicted output variable. Theta0 is the bias term and all the other theta values are coefficients. They are initiated randomly in the beginning, then optimized with the algorithm so that this formula can predict the dependent variable closely.
When theta values are initiated in the beginning, the formula is not trained to predict the dependent variable. The hypothesis is far away from the original output variable ‘Y’. This is the formula to estimate the cumulative distance of all the training data:
This is called the cost function. If you notice, it deducts y(the original output) from the hypothesis(the predicted output), takes the square to omit the negativity, sum up and divide by 2 times m. Here, m is the number of training data. You probably can see that cost function is the indication of the difference between the original output and the predicted output. The idea of a machine learning algorithm is to minimize the cost function so that the difference between the original output and the predicted output is closer. To achieve just that, we need to optimize the theta values.
Here is how we update the theta values. We take the partial differential of the cost function with respect to each theta value and deduct that value from the existing theta value,
Here, alpha is the learning rate and it is a constant. I am not showing the same formula for all the theta values. But It is the same formula for all the theta values. After the differentiation, the formula comes out to be:
This is called gradient descent.
The dataset I am going to use is from Andre Ng’s machine learning course in Coursera. I will provide the link at the bottom of this page. Please feel free to download the dataset and practice with this tutorial. I encourage you to practice with the dataset as you read if this is new to you. That is the only way to understand it well.
In this dataset, there are only two variables. But I developed the algorithm for any number of variables. If you use the same algorithm for 10 variables or 20 variables, it should work as well. I will use Numpy and Pandas library in Python. All these rich libraries of Python made the machine learning algorithm a lot easier. Import the packages and the dataset:
import pandas as pdimport numpy as npdf = pd.read_csv('ex1data2.txt', header = None)df.head()
Add a column of ones for the bias term. I chose 1 because if you multiply one with any value, that value does not change.
Add a column of ones for the bias term. I chose 1 because if you multiply one with any value, that value does not change.
df = pd.concat([pd.Series(1, index=df.index, name='00'), df], axis=1)df.head()
2. Define the input variables or the independent variables X and the output variable or dependent variable y. In this dataset, columns 0 and 1 are the input variables and column 2 is the output variable.
X = df.drop(columns=2)y = df.iloc[:, 3]
3. Normalize the input variables by dividing each column by the maximum values of that column. That way, each column’s values will be between 0 to 1. This step is not essential. But it makes the algorithm to reach it’s optimum faster. Also, if you notice the dataset, elements of column 0 are too big compared to the elements of column 1. If you normalize the dataset, it prevents column one from being too dominating in the algorithm.
for i in range(1, len(X.columns)):    X[i-1] = X[i-1]/np.max(X[i-1])X.head()
4. Initiate the theta values. I am initiating them as zeros. But any other number should be alright.
theta = np.array([0]*len(X.columns))#Output: array([0, 0, 0])
5. Calculate the number of training data that is denoted as m in the formula above:
m = len(df)
6. Define the hypothesis function
def hypothesis(theta, X):    return theta*X
7. Define the cost function using the formula of the cost function explained above
def computeCost(X, y, theta):    y1 = hypothesis(theta, X)    y1=np.sum(y1, axis=1)    return sum(np.sqrt((y1-y)**2))/(2*47)
8. Write the function for the gradient descent. This function will take X, y, theta, learning rate(alpha in the formula), and epochs(or iterations) as input. We need to keep updating the theta values until the cost function reaches its minimum.
def gradientDescent(X, y, theta, alpha, i):    J = []  #cost function in each iterations    k = 0    while k < i:                y1 = hypothesis(theta, X)        y1 = np.sum(y1, axis=1)        for c in range(0, len(X.columns)):            theta[c] = theta[c] - alpha*(sum((y1-y)*X.iloc[:,c])/len(X))        j = computeCost(X, y, theta)        J.append(j)        k += 1    return J, j, theta
9. Use the gradient descent function to get the final cost, the list of cost in each iteration, and the optimized parameters theta. I chose alpha as 0.05. But you can try with some other values like 0.1, 0.01, 0.03, 0.3 to see what happens. I ran it for 10000 iterations. Please try it with more or fewer iterations to see the difference.
J, j, theta = gradientDescent(X, y, theta, 0.05, 10000)
10. Predict the output using the optimized theta
y_hat = hypothesis(theta, X)y_hat = np.sum(y_hat, axis=1)
11. Plot the original y and the predicted output ‘y_hat’
%matplotlib inlineimport matplotlib.pyplot as pltplt.figure()plt.scatter(x=list(range(0, 47)),y= y, color='blue')         plt.scatter(x=list(range(0, 47)), y=y_hat, color='black')plt.show()
Some of the output points are almost overlapping with the predicted outputs. Some are close but not overlapping.
12. Plot the cost of each iteration to see the behavior
plt.figure()plt.scatter(x=list(range(0, 10000)), y=J)plt.show()
The cost kept going down with each iteration. This is the indication that the algorithm worked well.
I hope, this was helpful and you are also trying it for yourself. I encourage you to download the dataset and try running all the code for yourself if you are reading this to learn machine learning concepts. Here is the link to the dataset:
github.com
I hope it was helpful. Please feel free to follow me on Twitter and like my Facebook page.