Building Classification Models with Sklearn | by Sadrach Pierre, Ph.D. | Towards Data Science

Scikit-learn is an open-source machine learning library for python. It provides a variety of regression, classification, and clustering algorithms. In my previous post, A Brief Tour of Sklearn, I discussed several methods for regression using the machine learning package. In this post, we will go over some of the basic methods for building classification models. The documentation for this package is extensive and a fantastic resource for every data scientist. You can find the documentation here.
We will be using the Telco Customer Churn dataset. It can be found here
First, let’s import the data and print the first five rows:
import pandas as pd df = pd.read_csv("Customer_Churn.csv")print(df.head())
We will be using all of the categorical and numerical data to predict Churn. First, we need to convert the categorical columns into numerical values that the neural network can handle. For example, for gender we have:
df.gender = pd.Categorical(df.gender)df['gender_code'] = df.gender.cat.codes
Now, let’s define our input and output arrays:
import numpy as npfeatures = ['gender_code', 'SeniorCitizen_code', 'PhoneService_code', 'MultipleLines_code',                  'InternetService_code', 'Partner_code', 'Dependents_code', 'PaymentMethod_code',                  'PaymentMethod_code', 'PaperlessBilling_code','Contract_code', 'StreamingMovies_code',                 'StreamingTV_code', 'TechSupport_code', 'DeviceProtection_code', 'OnlineBackup_code',                 'OnlineSecurity_code', 'Dependents_code', 'Partner_code','tenure', 'MonthlyCharges']X = np.array(df[features])y = np.array(df['Churn_code'])
We will then split our data for training and testing:
from sklearn.model_selection import train_test_splitX_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 42)
Now all of our necessary variables are defined. Let’s build some models!
LOGISTIC REGRESSION
Let’s start with logistic regression. Logistic regression uses a logistic function to predict a binary dependent variable.
We import the LogisticRegression package as follows:
from sklearn.linear_model import LogisticRegression
Let’s define a logistic regression object, fit our model, and evaluate performance:
reg_log = LogisticRegression()reg_log.fit(X_train, y_train)y_pred = reg_log.predict(X_test)
We can visualize the predictions using metrics classification report:
from sklearn import metricsprint(metrics.classification_report(y_test, y_pred))
We can also look at the ‘roc_auc_score’ and the ‘f1_score.’ The ‘roc_auc_score’ is the area under the receiving operating characteristic curve. It is a measure of how well the binary classification model can distinguish classes. A ‘roc_auc_score’ of 0.5 means the model is unable to distinguish between classes. Values close to 1.0 correspond to a strong separation between classes. The ‘f1_score’ is the harmonic mean of precision and recall. Similar to ‘roc_auc_score’, a perfect ‘f1_score’ is equal to 1.0:
print("roc_auc_score: ", roc_auc_score(y_test, y_pred))print("f1 score: ", f1_score(y_test, y_pred))
RANDOM FORESTS
Now let’s take a look at random forests. Random forest is a tree-based method that ensembles multiple individual decision trees.
We import the RandomForestClassifier package as follows:
from sklearn.ensemble import RandomForestClassifier
Let’s define a random forest classification object, fit our model, and evaluate performance:
reg_rf = RandomForestClassifier()reg_rf.fit(X_train, y_train)y_pred = reg_rf.predict(X_test)
And let’s take a look at the metrics classification report:
print(metrics.classification_report(y_test, y_pred))
print("roc_auc_score: ", roc_auc_score(y_test, y_pred))print("f1 score: ", f1_score(y_test, y_pred))
We see that the random forest has worse performance than logistic regression. We can also print the feature importance. This allows us to see which variables are most significant for temperature prediction:
feature_df = pd.DataFrame({'Importance':reg_rf.feature_importances_, 'Features': features })print(feature_df)
I’d like to point out that by not passing any parameters, like max_depth and n_estimators, I selected default random forest values (which are n_estimators = 10 and max_depth = 10 ). We can further improve performance by optimizing parameters in random forests. This can be done manually or in an automated way using grid search techniques. I will leave the matter of parameter optimization for another post.
SUPPORT VECTOR MACHINES
The next method I’ll discuss is called support vector regression. This is an extension of support vector machines (SVM). SVMs construct a set of hyperplanes in high dimensional feature space that can be used for regression and classification problems.
We import the SVC package as follows:
from sklearn.svm import SVC
Let’s define a support vector classification object, fit our model, and evaluate performance:
reg_svc = SVC()reg_svc.fit(X_train, y_train)y_pred = reg_svc.predict(X_test)
We can visualize the predictions using metrics classification report:
print(metrics.classification_report(y_test, y_pred))
We can also look at the roc_auc_score and the f1_scores:
print("roc_auc_score: ", roc_auc_score(y_test, y_pred))print("f1 score: ", f1_score(y_test, y_pred))
We see that support vector classification performance is slightly worse than logistic regression and slightly better than random forests. Similar to the random forest, SVC takes parameters that can be used to optimize performance. These include the regularization parameter (default C = 1.0), kernel (default kernel = ‘rbf’) and the kernel coefficient (default gamma = ‘scale’)
K-NEAREST NEIGHBORS
The final method I will discuss is k-nearest neighbor for classification. K-nearest neighbors use Euclidean distance calculations where the prediction is the average of the k nearest neighbors.
We import the KNeighborsClassifier package as follows:
from sklearn.neighbors import KNeighborsClassifier
Let’s define a k-nearest neighbor classification object, fit our model, and evaluate performance:
reg_knn = KNeighborsClassifier()reg_knn.fit(X_train, y_train)y_pred = reg_knn.predict(X_test)
And let’s take a look at the metrics classification report:
print(metrics.classification_report(y_test, y_pred))
We can also look at the roc_auc_score and the f1_scores:
print("roc_auc_score: ", roc_auc_score(y_test, y_pred))print("f1 score: ", f1_score(y_test, y_pred))
The K-nearest neighbor algorithm also takes hyper-parameters, specifically n_neighbors, which can be selected to minimized error.
CONCLUSIONS
I will stop here but feel free to play around with model feature selection to see if you can improve the performance of some of these models.
To recap, I outlined a brief introduction to classification using the python machine learning library. I went over how to define model objects, fit models to data, and predict output using logistic regression, random forest, support vector machine, and K-nearest neighbor models.
I hope this post was informative. The code from this post is available on GitHub. Thank you for reading and happy machine learning!