A Summary of Logistic Regression
Tags: Sigmoid, Regression, Logistic Regression

Logistic Regression is one of the first Machine Learning algorithms most data scientist learn. In fact, logistic regression is the basis for Neural Networks, which are essentially networks of stacked logistic regressions!

Thus, it is crucial that anyone interested in data-science has a firm grasp of this method of classification. But first we need to understand the Sigmoid function.

Figure 1: The Sigmoid Activation function.

Figure 2: The non-linear sigmoid function, notice that for z=0, the function is 1/2.

Often, 0.5 is used as a cutoff value for logistic regression, where points that are above 0.5 being classified as 1, and points below it is as 0.

We can train the weights, or parameters, for logistic regression using gradient descent.

The sigmoid function is often used in classification and forms the basis of the softmax activation function. Notice another benifit of the sigmoid function, is that it is bounded between 0 and 1. Hence, unlike in linear regression, we cannot make a prediction greater than 1 or less than 0. This is perfect for classification problems.

The common Loss Function used for logistic regression is the Binary Cross Entropy. We also define the Cost function, which is simply the sum of the loss function over each training example , scaled by the number of training examples.

Consider a set of m training examples, where each column represents the predictors for the ith training example in a design matrix X of size p by m where p is the number of predictors and the outcomes in a matrix Y of size (1 by m).

Figure 3: The format we choose to consider. Columns of X are the observations, and rows are the predictors.

The common loss function in logistic regression is the binary cross entropy, and it takes the form

Figure 4: The loss function we use for classification.

Notice why this makes sense, if we get the prediction right, we have 0 loss as we get further and further away from the correct outcome, our loss grows.

Figure 5: We train weights by minimizing a loss function.

We want to train a set of weights for our predictors such that our predictions minimize the training loss.

Figure 6: Gradient Descent works by altering our weights depending on the derivative of the cost function.

We use gradient descent over the m training examples, the benefit of binary cross entropy is that it is convex, which ensures that we will converge towards a minimum. We always take steps towards the ideal set of weights. Note that alpha is the learning rate, and controls the step size we take in gradient descent.

Figure 7: The general process for applying gradient descent. The derivatives can be obtained by using the product and chain rule.

From Scratch:

We now implement the algorithm in python from scratch. We also vectorize our code so that it is efficient over large training sets.

To predict, we use our weights. If we are above 0.5, we predict True (or 1), otherwise we predict False (or 0).

We will illustrate using the divorce dataset from the UCI Irving ML library.

https://archive.ics.uci.edu/ml/datasets/Divorce+Predictors+data+set

This dataset consists of 170 observations, each with 54 predictors. The predictors are how the subject rated on a scale from 0 to 4 a series of questions regarding their marriage. The outcome ‘Class’ is divorced (1) or didn’t end up divorced (0).

We split our data into a training set and a test set. We will train our weights using the train data, and assess the model on the test data.

We can also plot the Cost as a function of iteration numbers, as suspected, the cost decreases with each iteration.

Figure 8: The Cost as a function of iterations. Our cost approaches zero quickly.

Check accuracy: In this simple example we used the a subset of the training data to fit the parameters and we end up being able to perfectly predict the test data after 15 iterations of gradient descent.