Comparing Classification Models for Wine Quality Prediction
By: Gulden Turgay
Tags: Data Science, Scikit Learn, Machine Learning, Exploratory Data Analysis, Random Forest

Validation and Model Selection

In this part I trained several classification algorithms to find the best one for the dataset I used.

K-Nearest Neighbors

I started with K-Nearest Neighbors classification algorithm. What this algorithm does is that it takes a data point and selects K number of observations in the training data which are the nearest to the data point and then predicts the response of the data point regarding the most popular response value from the K-nearest neighbors.

KNN accuracy plot

I created a plot to see how the accuracy changed with the number of K.

from sklearn.neighbors import KNeighborsClassifier

from sklearn.model_selection import cross_val_score # Number of k from 1 to 26

k_range = range(1, 26) k_scores = [] # Calculate cross validation score for every k number from 1 to 26

for k in k_range:

knn = KNeighborsClassifier(n_neighbors=k) # It’s 10 fold cross validation with ‘accuracy’ scoring

scores = cross_val_score(knn, X, y, cv=10, scoring=’accuracy’)

k_scores.append(scores.mean()) %matplotlib inline # Plot accuracy for every k number between 1 and 26

plt.plot(k_range, k_scores)

plt.xlabel('Value of K for KNN')

plt.ylabel('Cross-validated accuracy')

Cross Validation for KNN

I decided to go with k=19 since one of the highest accuracy obtained with it. And trained the model and calculated the accuracy with different validation methods.

# Train the model and predict for k=19 knn = KNeighborsClassifier(n_neighbors=19)

knn.fit(X_train, y_train)

y_pred = knn.predict(X_test) from sklearn.metrics import accuracy_score

from sklearn.metrics import roc_auc_score # classification report for test set

print(metrics.classification_report(y_test, y_pred, digits=3, zero_division = 1)) # Calculate cv score with 'accuracy' scoring and 10 folds

accuracy = cross_val_score(knn, X, y, scoring = 'accuracy',cv=10)

print('cross validation score',accuracy.mean()) # Calculate cv score with 'roc_auc_ovr' scoring and 10 folds

accuracy = cross_val_score(knn, X, y, scoring = 'roc_auc_ovr',cv=10)

print('cross validation score with roc_auc',accuracy.mean()) # Calculate roc_auc score with multiclass parameter

print('roc_auc_score',roc_auc_score(y_test,knn.predict_proba(X_test), multi_class='ovr'))

Classification table and validation scores for KNN

When I look at the classification report I immediately see that classes 4 and 8 have not been taken into consideration when training because their recall results are zero. This means, of all class 4 and class 8 members, it did not predict any of them correctly. So, it wouldn’t be a good model for our dataset.

Logistic Regression

Logistic regression is actually a binary classification algorithm which can be used for questions such as Yes/No, True/False, etc.

In this case it allows us to use it for multi-class classification problems such as ours. Because in our dataset there are 5 classes for quality to be predicted as. In order to use it as a multi-class classification algorithm, I used multi_class=’multinomial’, solver =’newton-cg’ parameters.

And considering it’s a multi-class classification problem, I used ‘roc_auc_ovr’ scoring parameter instead of ‘accuracy’ when calculating cross validation score. I also calculated roc_auc_score with multi_class=’ovr’ parameter. I will explain these later in conclusion.

# import module

from sklearn.linear_model import LogisticRegression # Train and fit model

logreg = LogisticRegression(multi_class=’multinomial’,solver =’newton-cg’)

logreg.fit(X_train, y_train) # Predict out-of-sample test set

y_pred = logreg.predict(X_test) # classification report

print(metrics.classification_report(y_test, y_pred, digits=3, zero_division = 1)) print(‘accuracy’,accuracy_score(y_test, y_pred)) # Calculate cv score with ‘roc_auc_ovr’ scoring and 10 folds

accuracy = cross_val_score(logreg, X, y, scoring = ‘roc_auc_ovr’,cv=10)

print(‘cross validation score with roc_auc’,accuracy.mean()) # Calculate roc_auc score with multiclass parameter

print(‘roc_auc_score’,roc_auc_score(y_test,logreg.predict_proba(X_test), multi_class=’ovr’))

Classification table and validation scores for logistic regression

Even though cross validation scores are a little bit higher, Some of the recall results are still zero. Let’s see what happens if we add some polynomial features.

Adding polynomial features to the logistic regression

from sklearn.preprocessing import PolynomialFeatures

from sklearn.pipeline import make_pipeline # Add polynomial features to the logistic regression model def PolynomialRegression(degree=2, **kwargs):

return make_pipeline(PolynomialFeatures(degree),

LogisticRegression(multi_class=’multinomial’,solver =’newton-cg’, **kwargs))

Now I tried adding 3rd degree polynomial features to the logistic regression model.

# Train and fit the 3rd degree polynomial regression model

poly = PolynomialRegression(3)

poly.fit(X_train,y_train) # Test out-of-sample test set

y_pred = poly.predict(X_test) # Classification report

print(metrics.classification_report(y_test, y_pred, digits=3)) # Calculate cv score with 'roc_auc_ovr' scoring and 10 folds

accuracy = cross_val_score(poly, X, y, scoring = 'roc_auc_ovr',cv=10)

print('cross validation score with roc_auc_ovr scoring',accuracy.mean()) # Calculate roc_auc score with multiclass parameter

print('roc_auc_score',roc_auc_score(y_test,poly.predict_proba(X_test), multi_class='ovr'))

Classification table and validation scores for 3rd degree polynomial regression

Finally, I had some representation for minority classes when predicting but they are very low and also cross validation score is lower than before.

Decision Tree

Decision trees are one of the most popularly used classification algorithms in the data science field. When I applied it to my dataset, there is an increase in recall results but the cross validation score decreased.

from sklearn.tree import DecisionTreeClassifier # Train and fit the Decision Tree Classification model

tree = DecisionTreeClassifier(random_state=0)

tree.fit(X_train, y_train) # Evaluate the model with out-of-sample test set

y_pred = tree.predict(X_test) # Classification report

print(metrics.classification_report(y_test, y_pred.round(), digits=3)) # Calculate cv score with ‘roc_auc_ovr’ scoring and 10 folds

accuracy = cross_val_score(tree, X, y,scoring = ‘roc_auc_ovr’,cv=10)

print(‘cross validation score with roc_auc_ovr scoring’,accuracy.mean()) # Calculate roc_auc score with multiclass parameter

print(‘roc_auc_score’,roc_auc_score(y_test,tree.predict_proba(X_test), multi_class=’ovr’))

Classification table and validation scores for decision tree

Random Forest

Random forest is an ensemble learning method that builds multiple decision trees and then gets a prediction based on what the majority of decision trees predict. I like to think of it as seeing multiple doctors for a health problem and deciding if you should get surgery or not depending on what the majority of the doctors say.

So let’s see the results of the random forest model.

from sklearn.ensemble import RandomForestClassifier # Train and fit the Random Forest Classification model

forest = RandomForestClassifier(n_estimators=100,random_state = 0)

forest.fit(X_train, y_train) # Test out-of-sample test set

y_pred = forest.predict(X_test) # Classification report

print(metrics.classification_report(y_test, y_pred.round(), digits=3)) # Calculate cv score with 'roc_auc_ovr' scoring and 10 folds

accuracy = cross_val_score(forest, X, y,scoring = 'roc_auc_ovr',cv=10)

print('cross validation score with roc_auc_ovr scoring',accuracy.mean()) # Calculate roc_auc score with multiclass parameter

print('roc_auc_score',roc_auc_score(y_test,forest.predict_proba(X_test), multi_class='ovr'))

Classification table and validation scores for decision tree

It’s the best so far! The roc_auc_score is pretty good, the cross-validation score is the best so far and there are some recall results even for the minority classes. But it’s not enough yet. So one of the things that can be done to increase recall is oversampling the minority classes. For this purpose I used the random forest algorithm with SMOTE algorithm implementation.

Adding SMOTE algorithm

SMOTE (Synthetic Minority Over-Sampling) algorithm creates synthetic minority class samples to increase the representation of minority classes.

# Import SMOTE module

from imblearn.over_sampling import SMOTE # Create model and fit the training set to create a new training set

sm = SMOTE(random_state = 2)

X_train_res, y_train_res = sm.fit_sample(X_train, y_train.ravel()) # Create random forest model

forest = RandomForestClassifier(n_estimators=100,random_state = 0) # Fit the model to the new train set

forest.fit(X_train_res, y_train_res.ravel()) # # Test out-of-sample test set

y_pred = forest.predict(X_test) # Classification report

print(metrics.classification_report(y_test, y_pred.round(), digits=3)) # Calculate cv score with 'roc_auc_ovr' scoring and 10 folds

accuracy = cross_val_score(forest, X, y,scoring = 'roc_auc_ovr',cv=10) print('cross validation score with roc_auc_ovr scoring',accuracy.mean()) # Calculate roc_auc score with multiclass parameter

print('roc_auc_score',roc_auc_score(y_test,forest.predict_proba(X_test), multi_class='ovr'))

Classification table and validation scores for the random forest with SMOTE

Even though the accuracy is almost the same as the previous one, the recall results have significantly increased for minority classes.