Logistic Regression for Binary Classification | by Sebastián Gerard Aguilar Kleimann | Towards Data Science

In previous articles, I talked about deep learning and the functions used to predict results. In this article, we will use logistic regression to perform binary classification. Binary classification is named this way because it classifies the data into two results. Simply put, the result will be “yes” (1) or “no” (0). To determine whether the result is “yes” or “no”, we will use a probability function:
This probability function will give us a number from 0 to 1 indicating how likely this observation will belong to the classification that we have currently determined to be “yes”. With this, we know what we intend to do with our prediction. Now, we’ll see how to use logistic regression to calculate the equation in (1).
To perform logistic regression, the sigmoid function, presented below with its plot, is used:
As we can see this function meets the characteristics of a probability function and equation (1). Likewise, we can see that when S(t) is very large positive, the function approaches one, and when S(t) is very large negative, the function approaches zero.
To be able to understand how logistic regression operates, we will make an example where our function will classify people as tall or not tall. The data that we will use to calibrate our function are those corresponding to the following table (Table 1):
Suppose t=wx+b, our goal will be to find w and b in such a way that by placing it in S(t) it will give us the correct prediction. We can start with a random number let’s say w=1 and b=0 and if S(t) is greater than 0.5 then we will consider it a tall person.
We notice that the parameters w=1 and b=0, don’t work since in all cases S(x)>0.5 Let’s try now with w=6 and b=-10.5, the result would be:
Great, we have found the w and b parameters in such a way that our function makes the predictions correctly!
These parameters work to make the prediction; however, many questions arise such as:
How do I calculate these parameters?
Are these parameters the most recommended?
To answer these questions, we will have to introduce two new topics that will help us optimize the function and understand the loss functions.
What are the best w and b parameters? The answer to this question is very simple because we want the parameters to give us as little error as possible. For this we use the loss error function:
The function basically tells us how far away is our estimate of the actual value (ŷ estimation, y actual value). If we’d make a summation on all our estimation we would get:
This summation is the total error of all our estimates and trying to reduce this function to 0 means trying to reduce our error to 0. However, when using this function on logistic regression we get a function that is not convex (we will return to this topic later) and since it is not convex there can be several local optimal points, and a great difficulty when calculating the best w and b. To solve this issue, we will use another function:
The function constructed in (5) has the same purpose as the function (3), to reduce the error. We’ll do an experiment on an observation to see how the function J(ŷ,y) behaves. Let’s imagine 4 possible scenarios of J(ŷ,y)
Scenario 1- y=1, ŷ=.99
We observe that in this scenario our estimation is practically correct and when replacing, we get:
Scenario 2- y=1, ŷ=.01
We observe that in this scenario our estimation is incorrect and when replacing, we get:
Scenario 3- y=0, ŷ=.99
We observe that in this scenario our estimation is incorrect and when replacing, we get:
Scenario 4- y=0, ŷ=.01
We observe that in this scenario our estimation is incorrect and when replacing, we get:
Intuitively, we can observe what this function does. Correct observations have a very low error, and incorrect observations have a very high error. If we did the summation on all the observations, we’d get.
The advantage of function (6) over function (4) is that it is convex. With function 6 it’s possible to find the optimal points in a simpler way using the gradient descent method.
The gradient descent method seeks to tell us in which direction we need to move our b and w parameters, to optimize the function and get the minimum error. The function described in (6) is convex so you could see it as the following graphic.
Now, suppose our loss function is J(w,b) and the parameters to be adjusted are (w,b). We want to find the point where J(w,b) is as small as possible. The gradient descent method tells us the direction where to move (w,b) to decrease J(w,b). These are the partial derivatives of (w,b), that is ∂J/∂w and ∂J/∂b.
Knowing the directions where to move w and b. We would only need to know the magnitude to which they would move, this is called a learning rate and is usually defined as α. To finally get:
Finally, we will summarize the steps that must be followed to perform the logistic regression:
Analyze the problem and accommodate the data.Propose w and b randomly to predict your data.Calculate the errorPerform gradient descent to get new w and b.
Analyze the problem and accommodate the data.
Propose w and b randomly to predict your data.
Calculate the error
Perform gradient descent to get new w and b.
These 4 steps should be iterated until you get an acceptable error.
We finally have all the theoretical elements to apply logistic regression. It is also important to know how to apply them in programming languages such as python. This language is widely used so the implementation of this algorithm is quite easy. For this example, we will use logistic regression to predict the trajectory of basketball players. The data obtained for this exercise was extracted from 1data.world. At the end of the article, the full code is displayed, so that you can follow and run the code along with the article on google colab.
It’s important to understand what each of the columns in this table mean:
Name- Name of the player
GP-Games played
Min-Minutes played
Pts-Average points made per game
FGM- Field goals made
FGA-Field goals attempted
%FG-Percentage of field goals
3P MADE- Three-pointers made
3PA- Three-pointers attempted
FTM-Free throws made
FTA-Free throws attempted
FT%-Free throw percentage
OREB-Offensive rebounds
DREB-Defensive rebounds
Reb-Rebounds
AST-Assists
STL-Steals
BLK-Blocks
TOV-Turnovers
TARGET_5YRS-If the player lasted 5 years or more, this value will be one, and it will be 0 otherwise.
Before logistic regression, observation and analysis of the data should be done. Pandas library is a very used library on python for handling data and we will use it to read and describe data.
##Import library and read dataimport pandas as pdnbalog=pd.read_csv("path_of_file")###See data descriptiondecri=nbalog.describe()
The code preceded by “#”are just comments, so with this code, we have done only 3 things:
Import the necessary libraries.Read the base file.Describe the existing data.
Import the necessary libraries.
Read the base file.
Describe the existing data.
The description indicates the quantity of data we have, the maximum value, minimum value, standard deviation, etc. By observing the data it can be seen that some fields are empty. Before training the model these issues have to be solved. The empty fields will be substituted by 0, and Pearson Correlation Coefficient will be used to observe the data with the highest correlation.
###Using the data described we notice that 3P% has some blank ###fields.These fields will be filled with 0. nbalog=nbalog.fillna(0) ###We check the correlation that exists between the data. pearson=nbalog.corr(method='pearson')
Using Pearson Correlation Coefficient we notice the columns with the highest correlation. This way the columns that are more useful should be kept and the others dropped.
###Some variables are higly correlated so they will be dropped ###(pearson>.8). nbalog=nbalog.drop(['MIN', 'PTS','FGM','3P Made','FTM','FTA','TOV','OREB','DREB'], axis=1)
With the clean data we can start training the model. For this, the library sklearn will be used. This library contains many models and is updated constantly making it very useful. In order to train the model, we will indicate which are the variables that predict and the predicted variable.
### X are the variables that predict and y the variable we are  ###trying to predict. X=nbalog[['GP','FGA','FG%','3PA','3P%','FT%','REB','AST','STL','BLK']] y=nbalog[['TARGET_5Yrs']]
Now, using the library sklearn the data will be divided in training and test set. With the training set, the model will be adjusted, and with the test set we will see how good the model is.
### The data has to be divided in training and test set. from sklearn.model_selection import train_test_split X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.25)
The previous piece of code divides the data between train and test set. The variable X is for the independent variables and y for the dependent variable. This time 75% of the set was used for training and 25% for testing. After separating the data it can be used to fit the model which in this case is the “LogisticRegression” model.
###We import the model that will be used. from sklearn.linear_model import LogisticRegression. # Create an instance of the model. logreg = LogisticRegression() # Training the model. logreg.fit(X_train,y_train) # Do prediction. y_pred=logreg.predict(X_test)
So, the model has been calibrated using the function .fit and it’s ready to predict using the test data. This is done using the function .predict and using the independent variables for testing (X_test). The results obtained can be compared with the real values (y_test) to see if it’s a good model.
# Analyzing the results. from sklearn import metrics cnf_matrix = metrics.confusion_matrix(y_test, y_pred)
The resulting matrix is known as confusion matrix. In the first quadrant the number of entries that were classified correctly with 0 are shown(61). The second and third quadrant sum the incorrect classification(99). Lastly, the fourth quadrant shows the classifications that were done correctly with number 1 (175). The accuracy can be calculated as follows:
print("Accuracy:",metrics.accuracy_score(y_test, y_pred))Output:.7045
With this, we conclude the article. As usual, I’ll leave you the code so you can test, run and try different models. Good day and congratulations on learning how to do logistic regression.
Bibliography
1Rippner, N. (2017, January 24). Binary Classification Exercise Dataset. Retrieved September 24, 2020, from https://data.world/.
2Indeed123 / CC BY-SA (https://creativecommons.org/licenses/by-sa/3.0)
3Lolikar / CC BY-SA (https://creativecommons.org/licenses/by-sa/4.0)
Originally published at https://datasciencestreet.com on September 25, 2020.