Implementing Regression With Gradient Descent From Scratch
By: Ashay Changwani
Tags: Numpy, Regression, Machine Learning

Logistic Regression

What’s the aim of linear regression? How does it work and why do we use it?

I’m sure these are fairly basic questions that people on the internet have explained much better than I will ever be able to.

So I’ll stick to the tricky parts. These parts have been documented as well, however, it’s best to try to understand the concepts yourself, and then implementing the code from scratch without any inspiration from anywhere.

The dataset we’ll be using is the publicly available iris dataset. Our job is to perform binary classification on the same.

The concept is simple: given a set of inputs, determine whether the output would be 0 or 1 (in case of binary classification).

How do we get the output to stick within the boundaries of 0 and 1?

The answer lies in the Sigmoid Function

Graph for the Sigmoid Curve

If you look at the graph, you’ll notice how irrespective of how large or small the input (x-axis) is, the output will always be within 0 and 1.

Talk is cheap, let’s get to the code.

def sigmoid(x):

return np.exp(x)/(np.exp(x)+1)

Great, so the output has been fixed. Given a set of inputs, we’ll be able to get the corresponding output.

But how do we map n number of features that would be present in the input dataset to a single value on the x-axis?

This is solved with a simple equation

Y = sigmoid(W * X + b)

Where Y represents our output, and W represents a weight vector of size (num_features,1), and X is the input vector of size (num_features, 1). The bias term is a scalar that is also trained along with the weights.

Clarification: for simplicity of the above equation, the dimensions of the actual equation have not been accounted for, but on accounting for them, the equation becomes:

Y = sigmoid (W.T * X + b)

The bias term, however, in logistic regression can often be ignored since by training the weights over a few iterations, it becomes redundant.

Great, so let’s write the function for it:

def pred_val(X, params):

return sigmoid(np.matmul(X, params))

Here, params represent the weight vector that was discussed earlier.

Next, to build a complete model, we’re going to need the Loss or Cost function. One approach could be to use the square root error, but that’s prone to local optima that would stop us from achieving a good rating on the test set.

The equation we’ll be using is fairly well-known, but if you want to learn why it’s so effective, this is a great resource.

L(y’,y) = - (y*log(y’) + (1-y)*log(1-y’))

Here y is the ground truth value and y’ is the predicted value.

When the ground-truth value (y) is 1, the cost function becomes:

L(y’,1) = -log(y’)

This is good because when y’ is 0 (prediction is incorrect), the cost function will be infinite (log 0 is negative ∞).

And when the ground-truth value (y) is 0, the cost function becomes:

L(y’,0) = -log(1-y’)

When y’ is 1(prediction is incorrect), the cost function ∞ again.

Let’s write the code for the same:

def cost_fn (X, y, params):

y_pred = pred_val(X, params)



return np.sum(-(y*np.log(y_pred) + (1-y)*np.log(1-y_pred)))

Finally, the forward propagation steps are completed. To implement the backpropagation section, we’re going to revise gradient descent for a quick sec.

Our values for W and b should minimize the cost function.

Our initial values for W and b will be 0 or random.

Then we’ll keep updating them according to the learning rate as well as the derivative of W and b which is dW and db respectively (It helps to think about the derivates as the slope of the convex function. If you’re at the extremes, the slope is high and the values for the derivates will be high. Therefore, the value of W and b will change by a large value. Conversely, as you approach the convex point, the values of derivates will decrease, and W and b will not change monumentally.)

We update W and b as:

W = W - alpha * dW

b = b - alpha * db

alpha is the learning rate. The higher it is, the more the values of the gradient can affect the new values. The values can be tweaked with trial and error since higher values can lead to rapid convergence in some cases, but the opposite in other cases.

The explanation for the derivations for dW and db is something I’d better leave to those better than me, but I’m sure a quick Google search would provide you with thorough reasoning.

The equation basically boils down to:

dZ = Y_Pred - Y

dW = X * dZ

db = dZ

We’ll run this section of updating the weights n_iter number of times, which represents the number of iterations we want to run the model through.

At every iteration, we’ll calculate the cost, which we’ll plot at the end to see how well our gradient descent is working.

Finally, our last segment is complete. Let’s write the function for it:

def gradient_descent(X, y, params, learning_rate,num_iters):



cost_history = np.zeros((num_iters, 1))

for i in range(num_iters):



y_pred = pred_val(X, params)



#dimensions of X = [num_samples, num_features]

#final dimensions have to be = [num_features, 1]

#transpose of X = [num_features, num_samples]

#dimensions of y_pred - y = [num_samples, 1]

#therefore need to premultiply the transpose of X with (y_pred - y)



derivative_of_cost = np.matmul(X.transpose(), y_pred - y)

derivative_of_cost *= learning_rate

derivative_of_cost /= len(y) #aka number of samples



params = params - derivative_of_cost



cost_history[i] = cost_fn(X, y, params)

return (cost_history, params)

The keen-eyed would notice that we’re passing all samples to the function at once, this is done to vectorize the code and make it run exponentially faster than using for loops. We’ve tweaked our derivates and cost to divide by the number of samples to achieve the mean for the same.

Finally, the model is complete! Let’s write some basic wrapper code and look at the results:

Graph of the cost vs the number of iterations

We can see that in the initial iterations itself, the model begins to converge pretty fast with a value of 0.05 as alpha or the learning rate.

Confusion matrix of results of the model

You can find the entire codebase for the implementation on my GitHub repo here.