How to tackle any classification problem end to end & choose the right classification ML algorithm. | by Shailaja Gupta | Towards Data Science

We are aware of many machine learning algorithms now. However when I was learning ML , I was unable to find an article covering all the steps of machine learning including data preprocessing, choosing an ML algorithms and then narrowing down on one for various classification problems. Hence I wrote this article so that anyone can find all the resources at one place. I have also covered the pros and cons of various ML algorithms in the article so that you know when to use which type of algorithm. You can simply copy paste the code templates given in this article and apply to any classification problem at hand. Let us begin :)
Download the dataset from here.About dataset: MNIST is a very famous dataset. This dataset contains 784 columns representing the pixels of a digit (Every digit is made up of 256 pixels on the x-axis and 256 pixels on the y-axis. 256 x 256 pixels = 784 pixels). It has a column label which identifies the digit from the 784 pixels.In the below article, we will classify a digit as 5 or not 5. We will thus deal with binary classification for the sake of simplicity. Also, it is seen that most of the classification problems are binary classification problems. Multi-class classification (classifying digits from 0 to 9) will be dealt with in another article.
Download the dataset from here.
About dataset: MNIST is a very famous dataset. This dataset contains 784 columns representing the pixels of a digit (Every digit is made up of 256 pixels on the x-axis and 256 pixels on the y-axis. 256 x 256 pixels = 784 pixels). It has a column label which identifies the digit from the 784 pixels.
In the below article, we will classify a digit as 5 or not 5. We will thus deal with binary classification for the sake of simplicity. Also, it is seen that most of the classification problems are binary classification problems. Multi-class classification (classifying digits from 0 to 9) will be dealt with in another article.
Handling missing valuesHandling non-numeric data (handling categorical, ordinal variables and strings)Feature engineering and creating new features out of existing features.Extracting only useful and relevant features: Feature selectionRemove features with missing values above a particular thresholdSplitting data into Training and test setsChecking for Class imbalance and tackling itStandard scaling of data
Handling missing values
Handling non-numeric data (handling categorical, ordinal variables and strings)
Feature engineering and creating new features out of existing features.
Extracting only useful and relevant features: Feature selection
Remove features with missing values above a particular threshold
Splitting data into Training and test sets
Checking for Class imbalance and tackling it
Standard scaling of data
Since MNIST dataset does not require much pre-processing(no missing values, no string or categorical features/columns etc.),we have taken a simple dataset known as housing dataset and illustrated most of the data pre — processing techniques using that dataset. In the dataset, we need to predict whether someone will default or not based on the learnings from the past data. The profile of people who were given loans in the past is given and also whether they defaulted or not.
Full code for data pre-processing of housing dataset is given below:
github.com
B.1. Handling missing values (housing dataset)
If we have missing values, then it is tough to implement most of the ML algorithms . Hence we need to handle them first. One way to do that is using Imputer (Imputer works like fillna).
There can be different type of missing values. For e.g. we could have missing string data, missing categorical data and numerical data.Hence we have first segregated the numeric and non-numeric columns and then filled their NAN values using different methods. NAN values of numeric columns are filled for instance here using median and NAN values of string columns are filled using ‘most_frequent’.
from sklearn.impute import SimpleImputercols_remove=['Name']  #col names which we wish to remove from modelX_train = X_train.drop(cols_remove, axis=1)cols = X_train.columns #new coumns without the columns removedX_train_numeric=X_train.select_dtypes(include=['int', 'float']).columnsX_train_non_numeric=X_train.select_dtypes(exclude=['int', 'float']).columns#joined columns numeric and non numericnew = X_train_numeric.tolist() + X_train_non_numeric.tolist()from sklearn.impute import SimpleImputerfrom sklearn.compose import ColumnTransformert = [('num', SimpleImputer(strategy='median'), X_train_numeric),('cat', SimpleImputer(strategy='most_frequent'), X_train_non_numeric)]transformer = ColumnTransformer(transformers=t, remainder='passthrough')X_train = transformer.fit_transform(X_train) #numpy array#DataFrame constructor with new columns names and added reindex for change by original orderX_train = pd.DataFrame(X_train, columns=new).reindex(cols, axis=1)
B.2. Handling non-numeric data (housing dataset)
All your input data (X_train or X_test data) must be numeric data before you can do classification! However, from your experience, you must know by now, that this will not always be the case. You will have non-numeric data too most of the times. Non-numeric data can be of 3 types: categorical, ordinal and string data. In the dataset of housing, type of job is ordinal data, gender is categorical data,Name here is string data. The difference between these types of data is:
Categorical/Nominal data: Gender: Male and Female are neither worse nor better here. They are just categories of people’s genders.
Ordinal data: Rating: There is a definite order among variables. Super skilled>Skilled>Unskilled.
String data: Name:This data is similar to categorical data in the sense that names are just labels. However there is a difference. In the case of categorical variables, there are only fixed number of categories. For the name field on the other hand, there will be many names and none of them may be the same.
Lets see how to deal with each of these data types:
B.2.1. Handling categorical data (variables do not have any order or rank and specify different categories.):
from sklearn.preprocessing import OneHotEncoderonehotencoder = pd.get_dummies(X['Gender'], drop_first=True)# Drop column non_numeric_col as it is now encodedX = X.drop('Gender',axis = 1)# Join the encoded dataframe to XX = X.join(onehotencoder)
B.2.2. Handling ordinal data: (Data where variables have a rank)
In the column ‘Type of job’, Super skilled>Skilled>Unskilled. We can tell the machine that one category is better than the other by assigning some numerical value to all these 3 labels. The higher the number the better it will be considered by ML Algorithms. Since we give 3 to Super skilled, 2 to skilled and 1 to unskilled; ML Algorithms will assume that Super skilled> Skilled> Unskilled , which is what we wanted.
X['Type of job'] = X['Type of job'].map( {'Super Skilled':3, 'Skilled':2, 'Unskilled':1})
B.2.3. Handling string data
In the above example, we have string data ‘Name’. The string data can in no way be converted into numbers. Can we extract some useful information from this field Name? The answer here is No. (however if we had some prefix before the name Mr., Miss, Mrs etc. we could have extracted some information from the field and hence worked on it, but right now no info can be extracted). Hence we remove the field from our modelling X_train.
#You can specify all the string columns you want to removecols_remove=['Name']X = X_train.drop(cols_remove, axis=1)
B.3. Feature engineering
Sometimes we can create new features from the given dataset or tweak existing features to extract more valuable information. All this requires domain knowledge. For example in the given dataset, we can create new features such as %Years employed and Credit income%. This gives an idea that how many years a person was actually working and what percentage of his income is credit. If he has more credit according to his income, then chances of default are more. If the person was working most of his life then, his chances of default are less.
X['Years employed percent'] = X['Years employed'] / X['Age']X['Credit Income percent'] = X['Amt of credit'] / X['Income']
B.4. Extracting only useful and relevant features: Feature selection
Having redundant and useless features in our model reduces accuracy. Let us see how to do feature selection for any dataset with hundreds of variables. Feature selection will remove the problem of overfitting from your classification models.
Though there are many methods for feature selection or variable reduction, we shall depict only few very effective and popular ways for feature selection.
B.4.1. If you need a specific number of features: SelectKBest class method.
It allows you to select top k most relevant features for your model(here we have taken k=5, so it will give us top 5 features).
#apply SelectKBest to extract top 5 best features for housing dfbestfeatures = SelectKBest(score_func=chi2, k=5)fit = bestfeatures.fit(X_train,y_train)dfscores = pd.DataFrame(fit.scores_)dfcolumns = pd.DataFrame(X_train.columns)#concat two dataframes for better visualization featureScores = pd.concat([dfcolumns,dfscores],axis=1)featureScores.columns = ['Features','Score']  #naming the dataframe columnsprint(featureScores.nlargest(5,'Score'))  #print 5best features
B.4.2. If you just want the most relevant and uncorrelated features: Boruta plot
In SelectKclass you need to specify that you want top k features. But sometimes, you do not know how many features you need. So you simply use Boruta plot. It will give you all the features which are important for your model. Also it will remove the highly correlated features, so that you will only have the most relevant features with you.
from sklearn.ensemble import RandomForestClassifierfrom boruta import BorutaPy# NOTE BorutaPy accepts numpy arrays only, if X_train and y_train #are pandas dataframes, then add .values attribute X_train.values in #that caseX_train = X_train.valuesy_train = y_train.values# define random forest classifier, with utilising all cores and# sampling in proportion to y labelsfrom sklearn.ensemble import RandomForestClassifierrf = RandomForestClassifier(n_jobs=-1, class_weight='balanced', max_depth=5)# define Boruta feature selection methodfeat_selector = BorutaPy(rf, n_estimators='auto', verbose=2, random_state=1)# find all relevant features - 5 features should be selectedfeat_selector.fit(X_train, y_train)# check selected features - first 5 features are selectedfeat_selector.support_# check ranking of featuresfeat_selector.ranking_# call transform() on X to filter it down to selected featuresX_filtered = feat_selector.transform(X_train)#To get the new X_train now with selected featuresX_train.columns[feat_selector.support_]
github.com
The full code for Boruta on MNIST dataset is mentioned above.
B.5. Remove features with missing values above a particular threshold
What if your column has 85% missing values? What should you do in that case? In that case it is better to drop the column. However sometimes business insists on keeping columns, then you can request for new data. What is the usually expected threshold above which we must not have missing values? Generally that threshold is expected at 25–30 %. Above this threshold, you should preferably drop the column, unless the customer insists on using it or if it is a very important parameter for analysis. In that case you can use imputer as depicted above.
#Obtain the sum of missing values in all columns df.isna().sum()#Find the percentage of missing values in df:result=df.isna().sum()/len(df)#In df remove columns where missing values are more than a thresholddf = df.loc[:,result > .8]
B.6. Splitting into Test and training data (MNIST Dataset)
We will split the train_df into test and training set using train_test_split to test our model’s accuracy before making predictions on the test_df:
X=train_df.drop('label', axis=1)y=train_df['label']# implementing train-test-splitfrom sklearn.model_selection import train_test_splitX_train, X_test, y_train_full, y_test_full = train_test_split(X, y, test_size=0.20, random_state=66)#Below part is only for MNIST dataset#For digit 5, output label=1, for any other digit output label=0y_train=((y_train_full==5).astype(int))  y_test=((y_test_full==5).astype(int))
B.7. Checking for Class imbalance and tackling it
Class imbalance means one class is dominating and there are very few instances of the other class. In MNIST dataset, there are only 3,060 instances where the digit is 5 out of around 40,000 total cases. That means our ML algorithm has exposure to very less training cases where the digit is 5.
y_train.value_counts()0    30540   #digit is not 51     3060   #digit is 5Name: label, dtype: int64
Class imbalance exists in our MNIST dataset as shown above.How to treat class imbalance?
Increase the minority class instances(class 1 in this case, the digit is 5) by using oversamplingReduce majority class instances(class 0 in this case, digit not 5) by using under-sampling.
Increase the minority class instances(class 1 in this case, the digit is 5) by using oversampling
Reduce majority class instances(class 0 in this case, digit not 5) by using under-sampling.
I personally prefer oversampling as it helps retain the variability of the data. SMOTE algorithm(Synthetic Minority Oversampling Technique) is a type of oversampling. At a high level, SMOTE:
Randomly choose one of the k-nearest-neighbours and using it to create a similar, but randomly tweaked new observations.
Randomly choose one of the k-nearest-neighbours and using it to create a similar, but randomly tweaked new observations.
We are going to implement SMOTE in Python.
from imblearn.over_sampling import SMOTEsm = SMOTE(random_state = 42)X_train_oversampled, y_train_oversampled = sm.fit_sample(X_train, y_train)X_train = pd.DataFrame(X_train, columns=X_train.columns)y_train=y_train_oversampled
pd.Series(y_train).value_counts().sort_index() #values after SMOTE0    305401    30540dtype: int64
After SMOTE implementation, both X_train and y_train have similar (in this case equal) number of values for both classes 0 and 1.
B.8. Standard scaling of data
from sklearn.preprocessing import StandardScalersc = StandardScaler()X_train = sc.fit_transform(X_train)X_test=sc.transform(X_test)
Test all/many classifiers for classification on training data.Select the one(s) that work best on training data and check their performance on test dataTry improvising those classifiers that work best by tuning their hyperparameters.
Test all/many classifiers for classification on training data.
Select the one(s) that work best on training data and check their performance on test data
Try improvising those classifiers that work best by tuning their hyperparameters.
Examples of classifiers which can be used for a classification project include Logistic Regression, K-Nearest Neighbors (KNN), SVM, Kernel SVM, Naive Bayes, Decision Tree classification, XGBoost and RandomForest classification to name a few. Treat these models as black-box models. You need not know the mathematics behind the models to implement them. Coding them is a simple plug and play according to the code given below.
C.1. Testing the model on various classification ML algorithms.
Let us train our model on Logistic Regression, K-Nearest Neighbors (KNN), SVM, Kernel SVM, Naive Bayes, Decision Tree classification, XGBoost and RandomForest classification one by one and test the performance on training and test data for each.
C.1.1. Applying Logistic Regression on MNIST Dataset
from sklearn.linear_model import LogisticRegressionlr_clf = LogisticRegression(random_state = 0)lr_clf.fit(X_train, y_train)
Logistic Regression performance on training and test data
print('Accuracy on training set:',lr_clf.score(X_train,y_train))print('Accuracy on test set:',lr_clf.score(X_test,y_test))
Accuracy on training set: 0.9622134905042568Accuracy on test set: 0.9460714285714286
Please note that lr_clf.score(X_test, Y_test) is equivalent to your print(classificationreport(Y_test, Y_pred)). But you do not need to calculate Y_pred; that is done internally by the library when you implement .score method.
C.1.2. Implementing Naive Bayes on MNIST Dataset
from sklearn.naive_bayes import GaussianNBnb_clf= GaussianNB()nb_clf.fit(X_train, y_train)
Naive Bayes performance on training and test data
print('Accuracy on training set:',nb_clf.score(X_train,y_train))print('Accuracy on test set:',nb_clf.score(X_test,y_test))
Accuracy on training set: 0.6189259986902423Accuracy on test set: 0.31214285714285717
C.1.3. Implementing RandomForest Regression on MNIST Dataset
from sklearn.ensemble import RandomForestRegressorrf_clf = RandomForestRegressor(n_estimators = 10, random_state = 0)rf_clf.fit(X_train, y_train)
Testing Random Forest Regression performance on training data
print('Accuracy on training set:',rf_clf.score(X_train,y_train))print('Accuracy on test set:',rf_clf.score(X_test,y_test))
Accuracy on training set: 0.9933483955468239Accuracy on test set: 0.818514583915758
C.1.4. Applying decision tree on MNIST dataset
from sklearn.tree import DecisionTreeClassifierdec_clf = DecisionTreeClassifier()dec_clf.fit(X_train, y_train)
Testing Decision Tree performance on training data
print(‘Accuracy on training set’,dec_clf.score(X_train, y_train))print('Accuracy on test set:',dec_clf.score(X_test, y_test))
Accuracy on training set 1.0Accuracy on test set: 0.9645238095238096
C.1.5. XGBoost:
'''Install xgboost following the instructions on this link:https://stackoverflow.com/questions/35139108/how-to-install-xgboost-in-anaconda-python-windows-platform. '''from xgboost import XGBClassifierxgb_clf= XGBClassifier()xgb_clf.fit(X_train, y_train)
XGBoost performance on training and test data
from sklearn.metrics import accuracy_scoreprint('Accuracy score on train_data: ', accuracy_score(y_true = y_train, y_pred = xgb.predict(x_train).round()))print('Accuracy score on test_data: ', accuracy_score(y_true = y_test, y_pred = xgb.predict(x_test).round()))
Accuracy score on train_data: 0.9498511904761905Accuracy score on test_data: 0.9333333333333333
C.1.6. Trying ensembles
Ensembles means that instead of using a single model, you take the the help of multiple models for prediction. For making each prediction, the ML ensemble algorithm takes every model’s output and suggestion into account and then makes a final prediction. If you take multiple models into account, then chances are that all of them make different errors. Hence when an ensemble takes multiple models’ predictions into consideration, one particular model’s error will have minimal effect on the overall result. Hence the overall prediction will be with minimal errors.
C.1.6.1. Ensemble of SVM and Logistic Regression
from sklearn.svm import SVCfrom sklearn.ensemble import VotingClassifiervoting_clf = VotingClassifier( estimators=[('lr', lr_clf), ('nb', nb_clf), ('decision', decision_clf)], voting='hard')voting_clf.fit(X_train, y_train)
Performance of Ensemble method using SVM and Logistic Regression
voting_clf.score(X_train, y_train)voting_clf.score(X_test, y_test)
Accuracy on training set 0.9787164374590701Accuracy on test set: 0.9401190476190476
voting_clf2 = VotingClassifier( estimators=[('lr', lr_clf),('decision', decision_clf)], voting='hard')voting_clf2.fit(X_train, y_train)
Performance of Ensemble method (Logistic regression & decision trees)
voting_clf2.score(X_test, y_test)voting_clf2.score(X_train, y_train)
Accuracy on training set 0.9777380952380952Accuracy on test set: 0.983513425016372
D. Classification Models that don’t work too well on large datasets
I am mentioning the below classification models since they are very useful for many classification problems that you will come across. They take too much time for implementation on MNIST dataset mainly due to the large size of MNIST. The performance of these algos below is too slow when the data is large, and hence it is not suggested to use them for MNIST.
k-NN
k-NN
k-NN works well with a small number of input variables/features/columns, but struggles when the number of inputs is very large. So in case the input variables are too many (784 here in MNIST), k-NN will not perform as well as the other algorithms.
2. Kernel SVM: The Kernel Matrix KK is a n×n matrix where, n is number of instances in the data. So for MNIST dataset, after the oversampling we have 60,000x784 instances ~ 470.4k, the kernel matrix KK will have (60,000x784 instances)2 ~ 2.2 x 1015 elements, and may take ~100G memories.
Basically, storing the kernel matrix requires memory that scales quadratically with the number of data points. Training time for traditional SVM algorithms also scales superlinearly with the number of data points. So, these algorithms aren’t feasible for large data sets.
Please find below how to implement these algorithms for other classification problems:
1.1. Applying k-NN on any classification dataset
from sklearn.neighbors import KNeighborsClassifierknn_clf = KNeighborsClassifier(n_neighbors = 5, metric = 'minkowski', p = 2)  #You can change these hyperparameters like metric etc.knn_clf.fit(X_train, y_train)
k-NN performance on training and test data of MNIST (time taken for output was close to 5 hours)
print('Accuracy on training set:',knn_clf.score(X_train,y_train))print('Accuracy on test set:',knn_clf.score(X_test,y_test))
Accuracy on training set 0.9742857142857143Accuracy on test set: 0.9907170923379175
1.2. Kernel SVM on MNIST Dataset
from sklearn.svm import SVCkernel_svm_clf = SVC(kernel = 'rbf', random_state = 0)kernel_svm_clf.fit(X_train, y_train)
Kernel SVM performance on training and test data (time taken for output was close to 5 hours)
print('Accuracy on training set:',kernel_svm_clf.score(X_train,y_train))print('Accuracy on test set:',kernel_svm_clf.score(X_test,y_test))
Accuracy on training set 1.0Accuracy on test set: 0.9125
While finalising a model, we need to keep both the accuracy and time taken in mind. While k-NN gives better accuracy than decision trees and random forest, it takes ages to compute. Despite working on Google colab with a GPU, I had to wait for 5 hours before getting output from both SVM and k-NN, hence we will not be using those methods for this dataset. Both k-NN and SVM are well suited for smaller datasets. We will rather choose models that do almost as well as SVM and k-NN and in fraction of time.
Based on the performances both time wise and accuracy wise, we shall go for ensemble of Logistic regression and decision trees. We chose to create the ensemble of LR and decision trees since both these models were individually doing the best. You can experiment with more ensembles too and check the accuracy of the models.
There are other parameters too such tuning the hyperparameters (hyperparameters are the parameters you input in each model, for e.g. in k-NN hyperparameters are specifying the k, for random forest, hyperparameters are specifying the number of trees etc.) inside each model which we could have done, but I will cover that in some other article.
If you are wondering about the pros and cons of various machine learning algorithms, and when to use which ML algorithm, their various applications, please check my article below.
medium.com
Happy reading :). Hope this helped :)