Calculus Behind Linear Regression | by Anuradha Wickramarachchi | Towards Data Science

Linear regression is usually the starting point for any machine learning course. The objective is to predict a linear relationship between an input variable to a target variable.
The naive case is the straight line that passes through the origin of space. Here we are limited to 2 dimensions in space, thus a cartesian plane. Let us develop gradually from ground up starting with y=mx format and then y=mx+c regression.
In this case, we know that we are going to fit a line to data that passes through the origin. This is the naive case which could be a good starting point. Let us develop our loss functions and see how it behaves. Let us rewrite our estimation as follows.
Note that we use the squared error divided by 2m where m is the number of data points we have. Therefore, we can consider this as a mean value. Precisely, this is the half of mean squared error. The intuition behind the division by 2 will be visible when we get the derivative of the loss function. This helps us to have a more simplified derivate for the loss function. Let us consider the following set of points.
I have generated these points using the y=5x graph with added noise. So ideally our estimation for θ should have a closer value for 5. Now if we plot the loss function for varying θ values we would get something like below.
Note that we have a minimum for the loss function closer to θ=5. Let us see how we can arrive at this particular minima computationally.
Note the orange coloured tangent line and its gradient. It is a positive gradient, which gives us the idea that we must go in the opposite direction to find a lower value.
The idea of gradient descent, as typically explained is to traverse downhill in an unknown terrain using only the knowledge of surrounding slopes. So in this example, the goal is to find the direction to traverse using the gradient of the slope and decide on the step size to move.
It is clear that we must go in the leftwards direction. What about the step size?. This is where the learning rate comes in to play. This is denoted as ⍺. Smaller the step, slower it is. However, taking a larger step could make you miss the minima (Imaging you step from θ=7 to θ=1, makes no sense). So we can formulate the change of θ as follows.
Since this scenario of linear regression is independent of the data, this can be used as a standard equation for computing loss and to do the gradient descent. This should emphasise the fact that we stick to standard loss functions always without making one of our own. It is very simple to implement this in python. We use len(x) for m and 0.001 as ⍺. x and y are the training data set or the set of points we want to estimate.
#Loss function(1/2* len(x)) * sum([(theta* x[i] - y[i])**2 for i in range(len(x))])#Change of thetatheta - 0.0001 * (1/len(x)) * sum([x[i]*(theta*x[i] - y[i]) for i in range(len(x))])
We can do the gradient descent iteratively until we meet a minimum or until the loss function is below a certain threshold. We can also decide on a number of iterations. Since we are guaranteed to have a minimum in linear regression, we can implement the descend as follows.
loss = float('inf')theta = 8while True:    theta = descend(theta)        if loss > loss_function(theta, x, y):        loss = loss_function(theta, x, y)    else:        break
The descend function return the next θ value and loss_function returns the loss at the value of θ. We could visualize the gradient traversal as follows.
This is the extension of the previous case and we can model the estimated equation and the loss function as follows.
Following are the set of points that I would try to fit a line. I have generated this using the function y=5 + 3x with a bit of noise.
Now that we have a loss function with 2 variables, our loss function will be a 3D plot with the third axis corresponding to the loss value. Diagrammatically, we could illustrate it as follows.
Our gradient descent can be similarly derived to reach the following set of equations.
Note that the gradients must be updated simultaneously so that update of one θ value will not affect the other. In python, this operation will look as follows.
theta1_new = theta1 - 0.01 * (1/len(x)) * sum([theta1 + theta2*x[i] - y[i] for i in range(len(x))])theta2_new = theta2 - 0.01 * (1/len(x)) * sum([x[i]*(theta1 + theta2*x[i] - y[i]) for i in range(len(x))])theta1, theta2 = theta1_new, theta2_new
The loss function is as follows. The only update from our previous scenario is that now we have the second variable θ2.
(1/2* len(x)) * sum([(theta1 + theta2* x[i] - y[i])**2 for i in range(len(x))])
I have updated my iteration through gradient descent as follows. In addition to the previous scenario of checking the decreasing loss, I consider the amount of drop-in loss to make sure I won’t iterate a lot closer to the minimum point.
loss = float('inf')theta1 = 10theta2 = -1while True:    theta1, theta2 = descend(theta1, theta2)        if loss > loss_function(theta1, theta2, x, y) and \            abs(loss - loss_function(theta1, theta2, x, y)) > 0.001:        loss = loss_function(theta1, theta2, x, y)    else:        break
We can visualize the descent over the gradient of the loss function as follows.
While our theta values are changing, our regression fit nicely comes in an agreement with the values on the table. It would look like the following.
We get the values 6.5 and 2.7 for the intercept and gradient respectively in the final fit which is reasonable given the noise in data.
Thanks for reading. Cheers!