Treating Linear Regression like More Than Just a Black Box
By: Ruben Winastwan
Tags: Statistics, Regression, Data Science, Machine Learning

Gradient Descent for Linear Regression

In a nutshell, Gradient Descent algorithm finds the optimum solution for a linear regression problem by tweaking model parameters θ (the bias term and the slope) to minimize the error or cost function with respect to training data iteratively.

Let’s visualize how Gradient Descent works to make it clearer.

Gradient Descent starts by randomly initializing an initial value of parameter value θ. This θ value then will be tweaked in every iteration such that the error or cost function is minimized. Once the Gradient Descent algorithm finds θ minimum, the cost function wouldn’t get smaller any further. This means that the algorithm has converged.

It is important to note that not all of the cost functions have a perfectly nice bowl shaped like in the image above. Most of the times, the cost function has all sorts of irregular shapes and plateaus like this:

The shape irregularity of the cost function makes it difficult for Gradient Descent algorithm to converge, as there is a huge chance that the algorithm will be stuck at the local optima and not finding the global optimum.

Fortunately, the cost function in a linear regression problem is always convex, meaning that there will be no local optima, just one global optimum solution. Thus, no matter how and where you initialize your initial value, it can ‘almost’ be guaranteed that the Gradient Descent algorithm will end up in the global minimum solution.

But the question is, why is it ‘almost’ be guaranteed that Gradient Descent will end up in the global minimum if there are no local optima?

Because there is one important parameter in Gradient Descent that you need to define in advance, which is the learning rate.

The learning rate determines how big is the step the algorithm should take in each iteration. If the learning rate is too small, there is a good chance that the global minimum wouldn’t be reached at the end of iterations. If the learning rate is too high, the algorithm will jump around the valley at each iteration and the algorithm will diverge.

Learning rate too small (left) and learning rate too high (right)

Implementation of Gradient Descent

After all of the explanation about Gradient Descent, let’s implement it with a code using the study case taxi fare vs travel distance.

For a linear regression problem, the cost function of Gradient Descent algorithm that we should minimize is the mean squared error (MSE).

where:

In the mathematical notation above, J is the cost function, m is the number of observations or data, hθ is the predicted value from linear regression, θ₀ is the bias term, θ₁ is the slope, x is the feature value and y is the actual predicted value. We can implement above equations in a code:

def computeCostFunction(x, y, theta, num_observations):



linear_function= np.matmul(x,theta)

error = np.subtract(linear_function,y)

error_squared = error.transpose()**2

cost_function = (0.5/num_observations)*(np.sum(error_squared))



return cost_function

The next important step is to compute the gradient of the cost function for each model parameter θⱼ. The intuition behind this gradient is how much the cost function will change if you tweak the value of parameter θⱼ.

For mean squared error, there is a closed form solution for its partial derivative. So, it is not a problem if you’re not an expert in Calculus.

Once you know the gradient of each parameter θⱼ, now you can update each parameter θⱼ for one iteration. This is where the learning rate α plays a role in a Gradient Descent algorithm.

Finally, we can implement the above equations in a code:

intercept_x = np.ones((len(x),1)) #add bias term

x = np.column_stack((intercept_x,x)) alpha = 0.05 #learning rate

iterations = 25000

theta = np.random.randn(2,1) #random initialization of parameters

num_observations = 1000 def gradientDescent(x, y, theta, alpha, num_observations, iterations):



cost_history = np.zeros((iterations,1))



for i in range (iterations):



linear_function= np.matmul(x,theta)

error = np.subtract(linear_function,y) gradient = np.matmul(x.transpose(),error)

theta = theta - (alpha * (1/num_observations) * gradient)



cost_history[i] = computeCostFunction(x, y, theta, num_observations)



return theta, cost_history theta, cost_history = gradientDescent(x, y, theta, alpha, num_observations, iterations) print(theta) #Output: array([[4.49781697],[1.98664487]])

The output from Gradient Descent looks the same from the one obtained with Scikit-learn. The difference is that you now know what’s under the hood.

There is one more straightforward approach to find the optimum solution for your linear regression problem, which is the Normal Equation.