ML From Scratch: Logistic and Softmax Regression | by Luke Newman | Towards Data Science

In this ML From Scratch series we create a library of machine learning algorithms in a similar style to Scikit-Learn’s using object-oriented programming. This means you can pip install the library and play around with all the available models in a style you are already familiar with.
If you have never implemented any learning algorithms from scratch, I’ll give you three good reasons to follow along and start now:
You’ll learn in great detail the special nuances within each model growing your understanding deeply.You’ll level up your coding skills by documenting code and creating test cases ensuring all functionality works as expected.You’ll develop a better understanding of object-oriented programming allowing you to improve your machine learning project workflow.
You’ll learn in great detail the special nuances within each model growing your understanding deeply.
You’ll level up your coding skills by documenting code and creating test cases ensuring all functionality works as expected.
You’ll develop a better understanding of object-oriented programming allowing you to improve your machine learning project workflow.
To view the Github repo visit here.
This project is meant to increase our understanding of learning algorithms and is not meant for use in actual data science work.
To use the classes and functions for testing purposes create a virtual environment and pip install the project.
    $ pip install mlscratch==0.1.0
To download all source code in a local repository create a virtual environment and run the following commands in your terminal.
    $ git clone https://github.com/lukenew2/mlscratch    $ cd mlscratch    $ python setup.py install
Logistic Regression is a common regression algorithm used in classification. It estimates the probability that an instance belongs to a particular class.
If the estimated probability is greater than or equal to 50%, the model predicts the instance belongs to the positive class. If the probability is less than 50%, the model predicts the negative class.
Logistic Regression works just like a Linear Regression model. It calculates a weighted sum of the feature array (plus a bias term), but instead of outputting the result directly like a Linear Regression model does, it outputs the logistic of this result.
The logistic — noted σ(*) — is a sigmoid function that transforms its input to a number between 0 and 1.
Once the model has estimated the probability that an instance belongs to the positive class, it simply makes its prediction based on the following rules.
For our implementation from scratch we’ll need to create a sigmoid function that can transform our inputs into probabilities. Let’s see how this is done.
Here we made a class and gave it one method. This special __call__ method let’s our class behave like a function when it is called. We’ll use this property soon when we create our Logistic Regression class.
Now that we know everything about how Logistic Regression estimates probabilities and makes predictions, let’s look at how it is trained.
The objective of training is to find the set of parameters Θ so that the model predicts high probabilities for positive instances and low probabilities for negative instances. This idea is captured by the cost function log loss.
Since we’ll be using Gradient Descent in our implementation, what we need to know is the partial derivatives of the cost function with regard to the model parameters Θ.
This equation is very similar to the cost function partial derivatives of Linear Regression. For each instance it computes the prediction error and multiplies it by the instance’s feature value, and then it computes the average over all instances.
When implementing Logistic Regression with batch gradient descent we don’t want to compute the average over all instances. We want the gradient vector containing all the partial derivatives allowing us to update our parameters in the opposite direction that they’re pointing.
Let’s get into the implementation. Please read the documentation to understand what each piece of code is doing.
There we go! We have a fully functional Logistic Regression model that can perform binary classification. Let’s write a test case to ensure its working properly.
In this test case we use the famous iris dataset and transform it into a binary classification problem, perform a train/test split, do some preprocessing, and then train our Logistic Regression model and test for accuracy greater than 80% on the test set.
Now, If we ever make changes to our code for whatever reason, we’ll always have a test case ensuring our changes didn’t break the model. Trust me, this saves a lot of headaches later on especially when our code base grows.
The Logistic Regression model we implemented only supports binary classification, but can be generalized to allow support for multiple classes. This is called Softmax Regression.
The idea is simple: for each instance, the Softmax Regression model computes a score for each class, then estimates the probability the instance belongs to each class by applying the softmax function to the scores.
In this equation:
K is the number of classes.
s(x) is a vector containing the scores of each class for the instance x.
Just like the Logistic Regression classifier, the Softmax Regression classifier predicts the class with the highest estimated probability.
Implementing the softmax function from scratch is a little tricky. When you divide exponents that can potentially be very large, you run into the issue of numerical stability. To avoid this, we use a normalization trick. Notice that if we multiply the top and bottom of the fraction by a constant C and push it into the sum, we get the following equivalent expression:
We are free to choose the value of C, but a common choice is to set log(C) equal to the negative of the max of the instance x. This shifts the values so the highest value is zero. Let’s see this in code:
Here we added a softmax class to the same module as our sigmoid class using a __call__ method so our class behaves like a function when called.
Now let’s take a look at training the Softmax Regression model and its cost function. The idea is the same as Logistic Regression. We want a model that predicts high probabilities for the target class, and low probabilities for the other classes. This idea is captured by the cost function cross entropy.
In this equation:
y is the target probability that the instance belongs to the target class. In our case, this value is always equal to 1 or 0, depending on whether the instance belongs to the class or not.
In the case of two classes cross entropy is equivalent to log loss.
Since we will be implementing Softmax Regression using Gradient Descent, we need the gradient vector of this cost function with regard to Θ.
When calculating the gradients, we need our target class (y) to be of the same dimension as our estimated probabilities (p). Our estimated probabilities will be of shape (n_samples, n_classes) because for each instance we’ll have a probability associated with that instance belonging to each class.
For example, if we have three classes labeled 0, 1, and 2 we need to transform our target vector containing ([1, 2, 0]) into an array that looks like this:
([[0, 1, 0],  [0, 0, 1],  [1, 0, 0]])
The first column represents class 0, the second column represents class 1, and the third column represents class 2. You might have noticed that we can accomplish this by One-Hot Encoding our target vector (y). Let’s see this in code:
Now we have every helper function we need to complete our Softmax Regression model. The code is well documented so please read for explanations of what each part is doing.
And that’s it! We have successfully added Softmax Regression to our machine learning library. Let’s take a quick look at a test case to ensure it is working properly.
We’ll use the iris dataset, but this time try to classify all three classes with at least 85% accuracy. First, we load the dataset, train/test split it, perform standardization, and then train our Softmax Regression model and predict the test set.
We’re finished! Our test case passed. We successfully added Softmax Regression to our machine learning library.
That concludes this part of the ML From Scratch series. We now have a machine learning library containing the most common linear models for regression and classification including:
Linear Regression
Polynomial Regression
Ridge
Lasso
Elastic Net
Logistic Regression
Softmax Regression
As always, thanks for reading and feedback is greatly appreciated!
In the next part of the series we’ll build upon our library and create support vector machines capable of classification and regression.
Give me a follow if you like the content you see here! :-)