Logistic Regression from Scratch. Learn how to build logistic regression... | by Arseny Turin | Towards Data Science

In this post, we’re going to build our own logistic regression model from scratch using Gradient Descent. To test our model we will use “Breast Cancer Wisconsin Dataset” from the sklearn package and predict if the lump is benign or malignant with over 95% accuracy. GitHub repo is here. So let's get started.
Essentially logistic regression model consists of two components: sigmoid function and features with weights:
The sigmoid function g(z) takes features and weights z as an input and returns a result between 0 and 1. The output of the sigmoid function is an actual prediction ŷ.
After the model made a prediction, we can evaluate the result with a cross-entropy loss function:
Natural logarithm works in our favor here, because it penalizes heavily if the prediction is far off the true value. For example, if the model predicted value ŷ=0.16 and true y=1, the error is high and vice versa:
The loss function consists of the two parts, but we can combine them into a single equation:
Here we added y and (1 - y) to cancel out one part over another depending on the output. We will use this function in our model to calculate loss and also in the Gradient Descent part of the model training.
Model training is essentially minimization of the loss function. We achieve that with the Gradient Descent technique, which can be broken into a few steps:
First, we find derivatives of the loss function with respect to each weight. Derivatives can tell which direction and by how much we should change weight to make the model loss a bit smaller.Updating each weight according to derivative until the local minimum is found, i.e. model doesn’t improve anymore so we can stop.
First, we find derivatives of the loss function with respect to each weight. Derivatives can tell which direction and by how much we should change weight to make the model loss a bit smaller.
Updating each weight according to derivative until the local minimum is found, i.e. model doesn’t improve anymore so we can stop.
This is the most crucial and hardest part. Without derivatives, the model wouldn’t train, so we will go into great detail in this part.
At this point we can drop summation function and focus on what's inside:
Notation: y — true value, ŷ — predicted value (sigmoid)
To find derivatives we’re going to use the chain rule:
We use the chain rule when we need to find a derivative of a function that contains another function and so on. In our case, we have a loss function that contains a sigmoid function that contains features and weights. So there are three functions down the line and we’re going to derive them one by one.
The derivative of the natural logarithm is quite easy to calculate:
Since we have two parts in the equation, we can derive them separately:
Now we can combine two parts back together and simplify:
This looks pretty good, now we focus on the sigmoid function.
According to the chain rule, we have to find the derivative of ŷ (sigmoid). You can find detailed explanations of it’s derivative on the internet because it’s used quite often in machine learning models, so I’ll just write the final result:
The last function in the chain is what contains in z — our features and weights. A derivative with respect to each weight (w) would be its feature value (x), for example: (x1 * w1)’ = x1, because derivative of w is 1.
According to the chain rule, we multiply each derived function, so we get the following:
We are done. In the end, we got quite a compact function which we will use in the Gradient Descent.
This part is the breeze comparing to finding derivatives. In programmers language, it’s simply for loop where we constantly updating weights.
In this process, we multiply each derived value by the learning rate parameter and then subtract this value from the weight:
# example of updating one weightepochs = 50lr = 0.1for _ in range(epochs):        w1 -= lr * x1 * (y_hat - y) # y_hat is predicted value
The learning rate is usually a small number to control how fast or slow we want to move towards minimization. Let’s take a look at the full code:
Breast Cancer Wisconsin Dataset 569 samples and 30 features:
After a train-test split model predicted malignant and benign lumps with 97% and 95% accuracy, which is a decent result.
The model we created could be used in real-life applications, and it’s great for educational purposes. Though sklearn LogisticRegression will work faster and more accurately, we can keep optimizing what we have and eventually achieve similar performance. Let me know in the comments if you have any questions or recommendations.
Thank you for reading,