Comparing Classification Models for Wine Quality Prediction | by Gulden Turgay | Towards Data Science

IntroductionExploratory AnalysisData PreprocessingValidation and Model SelectionConclusion
Introduction
Exploratory Analysis
Data Preprocessing
Validation and Model Selection
Conclusion
Here is my first ever blog post! Since I put myself in a continuous, never-ending learning process as a result of my decision of career transition from chemical engineering to data science, I decided to share what I learned and applied on projects as blog posts.
In this project I wanted to compare several classification algorithms to predict wine quality which has a score between 0 and 10. Since I like white wine better than red, I decided to compare and select an algorithm to find out what makes a good wine by using winequality-white.csv data sourced from the UCI Machine Learning Repository. The attributes in this dataset are:
fixed acidity
volatile acidity
citric acid
residual sugar
chlorides
free sulfur dioxide
total sulfur dioxide
density
pH
sulfates
alcohol
quality
Exploratory data analysis is a very important step to get an idea of what the dataset is like and what kind of modifications we need to make. I started with importing the libraries and modules and reading the data I’ll be using into pandas dataframe.
import matplotlib.pyplot as pltfrom scipy import statsimport seaborn as snsimport pandas as pdimport numpy as npfrom sklearn import metrics
white_wines = pd.read_csv('winequality/winequality-white.csv')
To understand the data, I took a look at the first couple of rows. It seems pretty clean but it’s 4898 rows of data and it wouldn’t be wise to check all the rows just by looking at it to see if there is a problem with the data. So I will check all the missing values at once in the next step.
print(white_wines.shape)white_wines.head()
Checking missing values
There is no missing value in the data so it’s ready to analyze but first outliers must be detected and eliminated before the modeling step.
white_wines.isnull().any()fixed acidity           Falsevolatile acidity        Falsecitric acid             Falseresidual sugar          Falsechlorides               Falsefree sulfur dioxide     Falsetotal sulfur dioxide    Falsedensity                 FalsepH                      Falsesulphates               Falsealcohol                 Falsequality                 Falsedtype: bool
Detecting outliers
I applied df.describe() method to get a sense of the descriptive statistics information of the dataset. It’s a very good method to have an intuitive understanding of the problems in the dataset. In a quick glimpse, I can see that there are some outliers.
Let’s have a look at the ‘residual sugar’ column. count says there are 4898 rows in this column. While mean and std means respectively the average and standard deviation values of the column, 25% of the values are under 1.70 and 75% of them are under 9.90. The interesting thing is while the average value is 6.39 and minimum value is 0.60, the maximum value is 65.80. It looks like an outlier.
white_wines.describe()
I used boxplot to visualize the distribution of the values in ‘residual sugar’ column to get a better insight. The actual maximum value is around 20 and the values bigger than that are outliers since they are not included in the box of observation.
sns.boxplot(white_wines[‘residual sugar’])
Eliminating outliers by using Z-score
Most of the time it’s important to remove outliers since they would most likely affect the performance of machine learning models. But let’s say 30% of your dataset are outliers. Then it may not be wise to remove them all because probably there is something more going on and it needs to be inspected further. In order to find and remove the outliers, I used z-score.
Its interpretation is take the data point or observation, subtract the mean of the population and divide it by the standard deviation. It represents how many standard deviations away a data point is from the mean. The data points which are too far from the mean are considered as outliers. In most cases threshold for outlier detection is either z-score>3 or z-score<-3. For this purpose I used zscore() function defined in SciPy library and set the threshold=3. After removing outliers there are 4487 rows left in the dataset which mean about 8.4% of the dataset has been removed as outliers.
z = np.abs(stats.zscore(white_wines))white_wines = white_wines[(z < 3).all(axis=1)]white_wines.shape(4487, 12)
Checking the correlation between attributes
When I checked the correlation between columns I can see that some of the features are strongly correlated with quality while some of them are not.
plt.subplots(figsize=(15, 10))sns.heatmap(white_wines.corr(), annot = True, cmap = ‘coolwarm’)
Checking the class imbalance
As a last step of exploratory data analysis I wanted to check the class imbalance. And it seems like there is a high class imbalance where the minority classes are less represented than the majority classes. This will be a crucial part of the modeling in the later steps.
white_wines[‘quality’].value_counts()6    20385    13097     8558     1614     124Name: quality, dtype: int64
In this step of the analysis I defined the features to train and test the machine learning model and the target to predict which is ‘quality’. And then I did standardization(also called z-score normalization) for the features because different scales of features may impact the performance of the machine learning models. For this purpose, I used StandardScaler() function defined in Scikit-learn. And finally I split the dataset into training and test sets 80% and 20% respectively.
Defining the features and the target
# Define features XX = np.asarray(white_wines.iloc[:,:-1])# Define target yy = np.asarray(white_wines[‘quality’])
Standardizing the dataset
from sklearn import preprocessingX = preprocessing.StandardScaler().fit(X).transform(X)
Train and test sets split
from sklearn.model_selection import train_test_splitX_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=0)print (‘Train set:’, X_train.shape, y_train.shape)print (‘Test set:’, X_test.shape, y_test.shape)Train set: (3589, 11) (3589,)Test set: (898, 11) (898,)
In this part I trained several classification algorithms to find the best one for the dataset I used.
I started with K-Nearest Neighbors classification algorithm. What this algorithm does is that it takes a data point and selects K number of observations in the training data which are the nearest to the data point and then predicts the response of the data point regarding the most popular response value from the K-nearest neighbors.
KNN accuracy plot
I created a plot to see how the accuracy changed with the number of K.
from sklearn.neighbors import KNeighborsClassifierfrom sklearn.model_selection import cross_val_score# Number of k from 1 to 26k_range = range(1, 26)k_scores = []# Calculate cross validation score for every k number from 1 to 26for k in k_range: knn = KNeighborsClassifier(n_neighbors=k)# It’s 10 fold cross validation with ‘accuracy’ scoring scores = cross_val_score(knn, X, y, cv=10, scoring=’accuracy’)  k_scores.append(scores.mean())%matplotlib inline# Plot accuracy for every k number between 1 and 26plt.plot(k_range, k_scores)plt.xlabel('Value of K for KNN')plt.ylabel('Cross-validated accuracy')
Cross Validation for KNN
I decided to go with k=19 since one of the highest accuracy obtained with it. And trained the model and calculated the accuracy with different validation methods.
# Train the model and predict for k=19knn = KNeighborsClassifier(n_neighbors=19)knn.fit(X_train, y_train)y_pred = knn.predict(X_test)from sklearn.metrics import accuracy_scorefrom sklearn.metrics import roc_auc_score# classification report for test setprint(metrics.classification_report(y_test, y_pred, digits=3, zero_division = 1))# Calculate cv score with 'accuracy' scoring and 10 foldsaccuracy = cross_val_score(knn, X, y, scoring = 'accuracy',cv=10)print('cross validation score',accuracy.mean())# Calculate cv score with 'roc_auc_ovr' scoring and 10 foldsaccuracy = cross_val_score(knn, X, y, scoring = 'roc_auc_ovr',cv=10)print('cross validation score with roc_auc',accuracy.mean())# Calculate roc_auc score with multiclass parameterprint('roc_auc_score',roc_auc_score(y_test,knn.predict_proba(X_test), multi_class='ovr'))
When I look at the classification report I immediately see that classes 4 and 8 have not been taken into consideration when training because their recall results are zero. This means, of all class 4 and class 8 members, it did not predict any of them correctly. So, it wouldn’t be a good model for our dataset.
Logistic regression is actually a binary classification algorithm which can be used for questions such as Yes/No, True/False, etc.
In this case it allows us to use it for multi-class classification problems such as ours. Because in our dataset there are 5 classes for quality to be predicted as. In order to use it as a multi-class classification algorithm, I used multi_class=’multinomial’, solver =’newton-cg’ parameters.
And considering it’s a multi-class classification problem, I used ‘roc_auc_ovr’ scoring parameter instead of ‘accuracy’ when calculating cross validation score. I also calculated roc_auc_score with multi_class=’ovr’ parameter. I will explain these later in conclusion.
# import modulefrom sklearn.linear_model import LogisticRegression# Train and fit modellogreg = LogisticRegression(multi_class=’multinomial’,solver =’newton-cg’)logreg.fit(X_train, y_train)# Predict out-of-sample test sety_pred = logreg.predict(X_test)# classification reportprint(metrics.classification_report(y_test, y_pred, digits=3, zero_division = 1))print(‘accuracy’,accuracy_score(y_test, y_pred))# Calculate cv score with ‘roc_auc_ovr’ scoring and 10 foldsaccuracy = cross_val_score(logreg, X, y, scoring = ‘roc_auc_ovr’,cv=10)print(‘cross validation score with roc_auc’,accuracy.mean())# Calculate roc_auc score with multiclass parameterprint(‘roc_auc_score’,roc_auc_score(y_test,logreg.predict_proba(X_test), multi_class=’ovr’))
Even though cross validation scores are a little bit higher, Some of the recall results are still zero. Let’s see what happens if we add some polynomial features.
Adding polynomial features to the logistic regression
from sklearn.preprocessing import PolynomialFeaturesfrom sklearn.pipeline import make_pipeline# Add polynomial features to the logistic regression modeldef PolynomialRegression(degree=2, **kwargs): return make_pipeline(PolynomialFeatures(degree), LogisticRegression(multi_class=’multinomial’,solver =’newton-cg’, **kwargs))
Now I tried adding 3rd degree polynomial features to the logistic regression model.
# Train and fit the 3rd degree polynomial regression modelpoly = PolynomialRegression(3)poly.fit(X_train,y_train)# Test out-of-sample test sety_pred = poly.predict(X_test)# Classification reportprint(metrics.classification_report(y_test, y_pred, digits=3))# Calculate cv score with 'roc_auc_ovr' scoring and 10 foldsaccuracy = cross_val_score(poly, X, y, scoring = 'roc_auc_ovr',cv=10)print('cross validation score with roc_auc_ovr scoring',accuracy.mean())# Calculate roc_auc score with multiclass parameterprint('roc_auc_score',roc_auc_score(y_test,poly.predict_proba(X_test), multi_class='ovr'))
Finally, I had some representation for minority classes when predicting but they are very low and also cross validation score is lower than before.
Decision trees are one of the most popularly used classification algorithms in the data science field. When I applied it to my dataset, there is an increase in recall results but the cross validation score decreased.
from sklearn.tree import DecisionTreeClassifier# Train and fit the Decision Tree Classification modeltree = DecisionTreeClassifier(random_state=0)tree.fit(X_train, y_train)# Evaluate the model with out-of-sample test sety_pred = tree.predict(X_test)# Classification reportprint(metrics.classification_report(y_test, y_pred.round(), digits=3))# Calculate cv score with ‘roc_auc_ovr’ scoring and 10 foldsaccuracy = cross_val_score(tree, X, y,scoring = ‘roc_auc_ovr’,cv=10)print(‘cross validation score with roc_auc_ovr scoring’,accuracy.mean())# Calculate roc_auc score with multiclass parameterprint(‘roc_auc_score’,roc_auc_score(y_test,tree.predict_proba(X_test), multi_class=’ovr’))
Random forest is an ensemble learning method that builds multiple decision trees and then gets a prediction based on what the majority of decision trees predict. I like to think of it as seeing multiple doctors for a health problem and deciding if you should get surgery or not depending on what the majority of the doctors say.
So let’s see the results of the random forest model.
from sklearn.ensemble import RandomForestClassifier# Train and fit the Random Forest Classification modelforest = RandomForestClassifier(n_estimators=100,random_state = 0)forest.fit(X_train, y_train)# Test out-of-sample test sety_pred = forest.predict(X_test)# Classification reportprint(metrics.classification_report(y_test, y_pred.round(), digits=3))# Calculate cv score with 'roc_auc_ovr' scoring and 10 foldsaccuracy = cross_val_score(forest, X, y,scoring = 'roc_auc_ovr',cv=10)print('cross validation score with roc_auc_ovr scoring',accuracy.mean())# Calculate roc_auc score with multiclass parameterprint('roc_auc_score',roc_auc_score(y_test,forest.predict_proba(X_test), multi_class='ovr'))
It’s the best so far! The roc_auc_score is pretty good, the cross-validation score is the best so far and there are some recall results even for the minority classes. But it’s not enough yet. So one of the things that can be done to increase recall is oversampling the minority classes. For this purpose I used the random forest algorithm with SMOTE algorithm implementation.
Adding SMOTE algorithm
SMOTE (Synthetic Minority Over-Sampling) algorithm creates synthetic minority class samples to increase the representation of minority classes.
# Import SMOTE modulefrom imblearn.over_sampling import SMOTE# Create model and fit the training set to create a new training setsm = SMOTE(random_state = 2) X_train_res, y_train_res = sm.fit_sample(X_train, y_train.ravel())# Create random forest modelforest = RandomForestClassifier(n_estimators=100,random_state = 0)# Fit the model to the new train setforest.fit(X_train_res, y_train_res.ravel())# # Test out-of-sample test sety_pred = forest.predict(X_test)# Classification reportprint(metrics.classification_report(y_test, y_pred.round(), digits=3))# Calculate cv score with 'roc_auc_ovr' scoring and 10 foldsaccuracy = cross_val_score(forest, X, y,scoring = 'roc_auc_ovr',cv=10)print('cross validation score with roc_auc_ovr scoring',accuracy.mean())# Calculate roc_auc score with multiclass parameterprint('roc_auc_score',roc_auc_score(y_test,forest.predict_proba(X_test), multi_class='ovr'))
Even though the accuracy is almost the same as the previous one, the recall results have significantly increased for minority classes.
In this project, I used K-Nearest Neighbors, Logistic Regression with polynomial features, Decision Tree, and Random Forest. With the roc_auc_score in Scikit-learn I calculated the AUC score for each model. Also using cross_val_score method I found AUC score using cross validation method by passing roc_auc_ovr parameter. Even though it's usually used for performance evaluation in binary classification, with the One-vs-Rest approach, I applied it to the multi-class classification problem. Evaluating the model with this method is advantageous when there is a high class imbalance. Also it does not require to set a classification threshold.
If we compare the cross validation scores and recall results of all models we can see that the best results were obtained with Random Forest Classifier with SMOTE method with 0.7565 cross validation score. Since we had highly imbalanced classes, SMOTE created synthetic minority sampling to balance the samples. And trained the model as if we had balanced classes in our dataset.
Even though we got the highest cross validation score which is around 0.75, it is still not an ideal model because the recall results are not representative enough for any of the classes. In this case, in order to improve the model we need more data to train it.
Thank you for reading!
If you have any feedback or correction, please don’t hesitate to reach out to me at my LinkedIn!