Python (Scikit-Learn): Logistic Regression Classification | by Maurizio Sluijmers | Towards Data Science

The process of differentiating categorical data using predictive techniques is called classification. One of the most widely used classification techniques is the logistic regression. For the theoretical foundation of the logistic regression, please see my previous article.
In this article, we are going to apply the logistic regression to a binary classification problem, making use of the scikit-learn (sklearn) package available in the Python programming language.
We will use the Titanic dataset (available on Kaggle), where the goal is to predict survival on the Titanic. That is, on the basis of the features (explanatory variables) included in the dataset, we want to predict whether a particular person survived the Titanic shipwreck.
We start by importing the required packages and loading the titanic data set.
import pandas as pdimport numpy as npimport matplotlib.pyplot as pltimport seaborn as snsfrom sklearn import metricsfrom sklearn.linear_model import LogisticRegressionfrom sklearn.model_selection import train_test_splittitanic = pd.read_csv(".../Titanic/train.csv")titanic.head()
Here, we see the different variables included in the data set. Let us briefly describe each variable:
PassengerId: The ID of the passenger;
Survived: The target variable; whether the person survived the shipwreck (1) or did not survive the shipwreck (0);
Pclass: The class the passenger was in on the ship (1, 2 or 3);
Name: The name of the passenger;
Sex: The gender of the passenger;
Age: The age of the passenger;
SibSp: Number of siblings and spouses on the ship;
Parch: Number of parents / children on the ship;
Ticket: Number of the ticket;
Fare: The fare of the ticket;
Cabin: Number of the cabin;
Embarked: Port of embarkation (C = Cherbourg, Q = Queenstown, S = Southhampton).
When applying any predictive algorithm, we can never use it immediately without having done any pre-processing of the data. This step is extremely important, and can never be overlooked. For this data set, we perform the following pre-processing steps:
1. Drop features that do not seem to add any value to our model
titanic.drop(['PassengerId', 'Name', 'Ticket', 'Cabin'], axis=1, inplace=True)
Here, we drop the PassengerId, Name, Ticket and Cabin features from the data set. The reason for this, is that these do not supply any predictive power to our model.
2. Create categorical dummies for the embarkment ports
ports = pd.get_dummies(titanic.Embarked, prefix='Embarked')ports.head()
We created three binary features from the categorical Embarked feature, since the model cannot handle the string names in the original categorical variable.
titanic = titanic.join(ports)titanic.drop(['Embarked'], axis=1, inplace=True)
3. Transform gender names to binaries
titanic.Sex = titanic.Sex.map({'male': 0, 'female': 1})
Transform the ‘male’ and ‘female’ names to binaries (0 and 1) through a mapping.
4. Replace missing values
titanic[pd.isnull(titanic).any(axis=1)]
Note that all the missing values (NaNs) appear to be in the Age feature. To solve for these missing values, let us fill the missing values with the average age in the data.
titanic.Age.fillna(titanic.Age.mean(), inplace=True)
Now that we have pre-processed the data, we can extract the explanatory variables in X and the target variable in y:
y = titanic.Survived.copy()X = titanic.drop(['Survived'], axis=1)
Next, we split the data in a training and test set. The training set is used to train the logistic regression model. The model learns from the features included in the training set. The test set is used to validate the performance of the logistic regression model. For each observation in the test set, we predict whether the person survived or not, and compare the predictions with the true values.
We split the data so that the training set consists of 75% of the data, and the test set consists of 25% of the data. We make use of the train_test_split module of the scikit-learn package.
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=123)
By making use of the LogisticRegression module in the scikit-learn package, we can fit a logistic regression model, using the features included in X_train, to the training data.
model = LogisticRegression()model.fit(X_train, y_train)
Next, now that we have trained the logistic regression model on the training data, we are able to use the model to predict whether the persons included in the test set survived the shipwreck:
y_pred = pd.Series(model.predict(X_test))y_test = y_test.reset_index(drop=True)z = pd.concat([y_test, y_pred], axis=1)z.columns = ['True', 'Prediction']z.head()
Here, we see that for the first five observations of the test set, the logistic regression model predicted 4 out of 5 correctly.
To evaluate the entire test set, we can use the metrics module from the scikit-learn package.
print("Accuracy:", metrics.accuracy_score(y_test, y_pred))print("Precision:", metrics.precision_score(y_test, y_pred))print("Recall:", metrics.recall_score(y_test, y_pred))
Output:
Accuracy: 0.8026905829596412Precision: 0.7631578947368421Recall: 0.6904761904761905
That is, the logistic regression model results in 80.3% accuracy. Definitely not bad for such a simple model!
Of course, the model performance could be further improved by e.g. conducting further pre-processing, feature selection and feature extraction. However, this model forms a solid baseline.
A nice way to visualize the results of the model is by making use of a confusion matrix. Here, we use matplotlib and seaborn to create a nice confusion matrix figure.
cnf_matrix = metrics.confusion_matrix(y_test, y_pred)labels = [0, 1]fig, ax = plt.subplots()tick_marks = np.arange(len(labels))plt.xticks(tick_marks, labels)plt.yticks(tick_marks, labels)# create heatmapsns.heatmap(pd.DataFrame(cnf_matrix), annot=True, cmap="YlGnBu", fmt='g')ax.xaxis.set_label_position("top")plt.title('Confusion matrix', y=1.1)plt.ylabel('True')plt.xlabel('Predicted')
In the confusion matrix, we see that 121 persons that did not survive the shipwreck were correctly predicted as not surviving, 18 persons that did not survive the shipwreck were incorrectly predicted as surviving the shipwreck, 26 persons that survived the shipwreck were incorrectly predicted as not surviving the shipwreck and lastly, 58 persons that survived the shipwreck were correctly predicted.
Pros:
Does not require high computation power;
Easy to implement;
Straightforward interpretability.
Cons:
Vulnerable to overfitting;
Cannot solve non-linear problems;
Not able to handle a large number of categorical features.
Hopefully, after reading this article, you are now able to utilize the logistic regression technique yourself by making use of the scikit-learn package within Python. Good luck on using it within your own projects!