Building ML Model to Predict Whether the Cancer Is Benign or Malignant on Breast Cancer Wisconsin Data Set !! Part 4
Tags: Grid Search, Random Search, Hyperparameter Tuning, Scikit Learn, Deployment

Building ML Model to Predict Whether the Cancer Is Benign or Malignant on Breast Cancer Wisconsin Data Set !! Part 4

In part 1 of this series, we understood in depth the data health check up, and Exploratory Data Analysis using visualization techniques. In part 2, we understood the different kind of feature selection techniques. Also we understood the voted method. In Part 3, we built seven different models for each feature selection methods and analyzed & compared the performance of the models.

Now, in this final part 4, we will understand the Step 8–9 and will try to tune the hyperparameters of the seven different models (1. Logistic Regression, 2. Random Forest Classifier, 3. Gradient Boosting Classifier, 4. Extra Trees Classifier, 5. XGB Classifier, 6. KNeighbors Classifier and 7. SVM Classifier) using two different optimization approaches (1. Random and 2. Grid Search) and then we will discuss the model deployment options.

Steps involved in doing Machine Learning Model to predict whether the cancer is benign or malignant:

Step 1: Define Problem Statement

Step 2: Data Source

Step 3: Cleaning the Data

Step 4: Data Analysis and Exploration

Step 5: Feature Selection

Step 6: Data Modeling

Step 7: Model Validation

Step 8: Hyperparameter Tuning

Step 9: Deployment In this part 4, we will cover steps 8 & 9. We are using the Breast Cancer Wisconsin dataset available on UCI Machine Learning Repository.

Problem Statement

Our objective is to identify which features are most helpful in predicting malignant or benign cancer and to classify whether the breast cancer is benign or malignant.

Data Source

We have used publicly available dataset Breast Cancer Wisconsin and have download from UCI Machine Learning Repository. Repository: https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+%28Diagnostic%29.

The data, in csv format, can be downloaded from this link

At this Github link, you can access all the code and data of the project.

Step 8: Hyperparameter Tuning

1. Tuning or optimizing hyperparameters using RandomizedSearchCV

Using Scikit-Learn’s RandomizedSearchCV method, we can define a grid of hyperparameter ranges, and randomly sample from the grid, performing K-Fold CV with each combination of values.

Define a function to load the data



def LoadData():

global feature_names, response_name, n_features, model_full

model_full = d.read_csv('

# we change the class values (at the column number 2) from B to 0 and from M to 1

model_full.iloc[:,1].replace('B', 0,inplace=True)

model_full.iloc[:,1].replace('M', 1,inplace=True)

response_name = ['diagnosis']

drop_list = ['Unnamed: 32','id','diagnosis']

model_full_x= model_full.drop(drop_list,axis = 1)

X = model_full_x

y = model_full.diagnosis



X_train, X_test, y_train, y_test = train_test_split(X,y,

test_size = 0.3,

random_state = 12345)

return X_train, y_train, X_test, y_test from sklearn.model_selection import train_test_splitdef LoadData():global feature_names, response_name, n_features, model_fullmodel_full = d.read_csv(' https://raw.githubusercontent.com/Muhd-Shahid/Breast-Cancer-Wisconsin/master/data_breast-cancer-wiscons.csv' # we change the class values (at the column number 2) from B to 0 and from M to 1model_full.iloc[:,1].replace('B', 0,inplace=True)model_full.iloc[:,1].replace('M', 1,inplace=True)response_name = ['diagnosis']drop_list = ['Unnamed: 32','id','diagnosis']model_full_x= model_full.drop(drop_list,axis = 1)X = model_full_xy = model_full.diagnosisX_train, X_test, y_train, y_test = train_test_split(X,y,test_size = 0.3,random_state = 12345)return X_train, y_train, X_test, y_test

Tree based feature selection with random forest classifier

Here, we used the same optimal feature sets that we selected in Part 2 because we want to compare optimized model with previous one. you can also try with all features and see the performance of the model.

# 5. Tree based feature selection with random forest classification

fs_rf = ['texture_mean', 'area_mean', 'concavity_mean', 'area_se', ‘concavity_se’, ‘fractal_dimension_se’, 'smoothness_worst','concavity_worst', 'symmetry_worst',

'fractal_dimension_worst']

Scaling the data using StandardScaler()

x_train, y_train, x_test, y_test = LoadData()

from sklearn.preprocessing import StandardScaler

# Included the relevant features

x_train = x_train[fs_rf]

x_test = x_test[fs_rf] # scaling data

scale = StandardScaler()

X_train = scale.fit_transform(x_train)

X_test = scale.fit_transform(x_test)

1.1 Define Hyperparameter Grid for RandomizedSearchCV

To use RandomizedSearchCV, here, we have defined dictionary of models and a dictionary of parameters for each of the models for the helper class.

from sklearn.ensemble import ExtraTreesClassifier

from sklearn.ensemble import RandomForestClassifier

from sklearn.linear_model import LogisticRegression

from sklearn.ensemble import GradientBoostingClassifier

from sklearn.svm import SVC

from sklearn.neighbors import KNeighborsClassifier models1 = {

'Logistic Regression' : LogisticRegression(),

'RandomForestClassifier': RandomForestClassifier(),

'GradientBoostingClassifier': GradientBoostingClassifier(),

'ExtraTreesClassifier': ExtraTreesClassifier(),

'KNeighbors Classifier' : KNeighborsClassifier(),

'SVC': SVC()

}

params1 = {

‘Logistic Regression’ : {‘penalty’ : [‘l1’,’l2'], ‘C’ : np.logspace(0, 4, 10), ‘solver’ :[‘liblinear’]},

‘RandomForestClassifier’: { ‘n_estimators’: [16, 32] },

‘GradientBoostingClassifier’: { ‘n_estimators’: [16, 32], ‘learning_rate’: [0.8, 1.0] },

‘ExtraTreesClassifier’: { ‘n_estimators’: [16, 32] },

‘KNeighbors Classifier’ : {‘n_neighbors’ :[1,2,5,6,7,10], ‘weights’:[‘uniform’, ‘distance’]},

‘SVC’: {‘kernel’: [‘linear’,’rbf’], ‘C’: [1, 10], ‘gamma’: [0.001, 0.0001]}

}

On each iteration, the algorithm will choose a difference combination of the features. combination, but selecting at random to sample a wide range of values.

1.2 Training of the models for RandomizedSearchCV

The credit for the helper class code below goes to Panagiotis Katsaroumpas who initially wrote it, the original version is on this blog post.

import pandas as pd

import numpy as np

from sklearn.model_selection import RandomizedSearchCV class EstimatorSelectionHelper: def __init__(self, models, params):

if not set(models.keys()).issubset(set(params.keys())):

missing_params = list(set(models.keys()) - set(params.keys()))

raise ValueError("Some estimators are missing parameters: %s" % missing_params)

self.models = models

self.params = params

self.keys = models.keys()

self.grid_searches = {} def fit(self, X, y, cv=5, n_jobs=3, verbose=1, scoring=None, refit=False):

for key in self.keys:

print("Running RandomizedSearchCV for %s." % key)

model = self.models[key]

params = self.params[key]

gs = RandomizedSearchCV(estimator= model, param_distributions = params, cv=cv, n_jobs=n_jobs,

verbose=verbose, scoring=scoring, random_state = 2020)

gs.fit(X,y)

print('===========================================================')

print('Best Score: %s' % gs.fit(X,y).best_score_)

print('Best Hyperparameters: %s' % gs.fit(X,y).best_params_)

print('===========================================================')

self.grid_searches[key] = gs def score_summary(self, sort_by='mean_score'):

def row(key, scores, params):

d = {

'estimator': key,

'min_score': min(scores),

'max_score': max(scores),

'mean_score': np.mean(scores),

'std_score': np.std(scores),

}

return pd.Series({**params,**d}) rows = []

for k in self.grid_searches:

print(k)

params = self.grid_searches[k].cv_results_['params']

scores = []

for i in range(self.grid_searches[k].cv):

key = "split{}_test_score".format(i)

r = self.grid_searches[k].cv_results_[key]

scores.append(r.reshape(len(params),1)) all_scores = np.hstack(scores)

for p, s in zip(params,all_scores):

rows.append((row(k, s, p))) df = pd.concat(rows, axis=1).T.sort_values([sort_by], ascending=False) columns = ['estimator', 'min_score', 'mean_score', 'max_score', 'std_score']

columns = columns + [c for c in df.columns if c not in columns] return df[columns]

1.3 Evaluation of the models for RandomizedSearchCV

# Define the scoring

scoring_strings = ['accuracy', 'precision', 'recall', 'f1', 'average_precision','roc_auc']

scores_all = pd.DataFrame()

# Create object of the class helper

helper1 = EstimatorSelectionHelper(models1, params1)

flag=True

for scoring in scoring_strings:

helper1.fit(X_train, y_train, cv=5, scoring=scoring, n_jobs=2)

df_scores = helper1.score_summary(sort_by='max_score')

if flag:

# Saving each model for each scoring. Plz change with ur path

df_scores.to_csv(".../Breast Cancer Wisconsin/Code/Optimization/random/"+scoring+".csv")

df_scores['scoring_name']=scoring

scores_all = df_scores

print(scoring)

print(scores_all)

flag=False

else:

# Saving each model for each scoring. Plz change with ur path

df_scores.to_csv(".../Breast Cancer Wisconsin/Code/Optimization/random/"+scoring+".csv")

df_scores['scoring_name']=scoring

scores_all= scores_all.append(df_scores, ignore_index=True, sort=False)

print(scoring)

print(scores_all)



# Saving each model for each scoring. Plz change with ur path

scores_all.to_csv(".../Breast Cancer Wisconsin/Code/Optimization/random/scores_all.csv")

1.4 Review the outcomes of RandomizedSearchCV

Below outcome is the heighest mean scores of the each scoring (accuracy, average_precision, f1, precision, recall, roc_auc) for each algorithms. Also, added the working sheet of all scores excel sheet(scores_all_random.xlsx) for your understanding.

2. Tuning or optimizing hyperparameters using GridSearchCV

2.1 Define Hyperparameter Grid for GridSearchCV

params1 = {

'Logistic Regression' : {'penalty' : ['l1','l2'], 'C' : np.logspace(0, 4, 10), 'solver' :['liblinear']},

'RandomForestClassifier': { 'n_estimators': [16, 32] },

'GradientBoostingClassifier': { 'n_estimators': [16, 32], 'learning_rate': [0.8, 1.0] },

'ExtraTreesClassifier': { 'n_estimators': [16, 32] },

'KNeighbors Classifier' : {'n_neighbors' :[1,2,5,6,7,10], 'weights':['uniform', 'distance']},

'SVC': {'kernel': ['linear','rbf'], 'C': [1, 10], 'gamma': [0.001, 0.0001]}

}

2.2 Training of the models for GridSearchCV

I used the same EstimatorSelectionHelper class and changed the GridSearchCV() function as below.

from sklearn.model_selection import GridSearchCV gs = GridSearchCV(model, params, cv=cv, n_jobs=n_jobs,

verbose=verbose, scoring=scoring, refit=refit,

return_train_score=True)

2.3 Evaluation of the models for GridSearchCV()

For evaluation of the models, I used the same code for each scoring.

for scoring in scoring_strings:

helper1.fit(X_train, y_train, cv=5, scoring=scoring, n_jobs=2)

df_scores = helper1.score_summary(sort_by='max_score')

2.4 Review the outcomes of GridSearchCV()

Below outcome is the heighest mean scores of the each scoring (accuracy, average_precision, f1, precision, recall, roc_auc) for each algorithms. Also, added the working sheet of all scores excel sheet(scores_all_grid.xlsx) for your understanding. We are getting the same performance scores and same parameters for both the approaches.

Comparisons

We can make some quick comparisons between the different approaches used to improve performance showing the returns on each. The following table shows the final results from all the improvements we made (including those from the third part without hyperparameter search). From the below table, it is clear that there is no improvement if scoring is ROC_AUC and if scoring is accuracy then there is slight improvement):

Limitations

You may get better results with more combinations of hyperparameter grid You may get better results with different split of the data. Used only one set of features(tree based features) Did not check the testing performance

Step 9: Model Deployment

Choosing how to deploy a predictive models into production is quite a complex affair, there are different way to handle the lifecycle management of the predictive models, different formats to stores them, multiple ways to deploy them and very vast technical landscape to pick from. Here, just giving you the idea for the deployment.

There are mainly two different models of model deployment that are Batch Mode and Real-time Mode.

9.1 Batch Mode example

from sklearn.linear_model import LogisticRegression

import sklearn.model_selection

import pickle

import os clf_lr = LogisticRegression(C=7.7426,penalty='l2',solver='liblinear')

clf_lr.fit(X, y)

os.chdir('/Users/Shahid/Desktop/deployment/')

pickle.dump(cl_lr, open('lr_model', 'wb'))

load_model = pickle.load(open(filename, 'rb'))

result = load_model.predict(X_unseenData)

print(result)

9.2 Real-time Mode

We will discuss deploying ML Models using Flask API in our next post.

Conclusions

Model Hyperparameter tuning is very useful to improve the performance of a ML model. We have discussed both the approaches to do the tuning that is GridSearchCV and RandomizedSeachCV across the multiple models. The only difference between both the approaches is in grid search we define the combinations and do training of the model whereas in RandomizedSearchCV the model selects the combinations randomly. Both are very effective ways of tuning the parameters that increase the model generalizability. Also, we discussed little bit about model deployment.

End Notes

This ends our Part 4 on model building and evaluation. The aim of this Part 4 was to provide step by step guide to use different approaches to tune the hyperparameter and model deployment.

Personally, I enjoyed writing this and would love to learn from your feedback. Did you find this Part 4 useful? I would appreciate your suggestions/feedback. Please feel free to ask your questions through comments below.

All the code and datasets used in this article can be accessed from my GitHub.

Thanks for reading :)

Resources