Wine data set: A Classification Problem | by Ali Faghihnejad | Towards Data Science

The wine data set consists of 13 different parameters of wine such as alcohol and ash content which was measured for 178 wine samples. These wines were grown in the same region in Italy but derived from three different cultivars; therefore there are three different classes of wine. The goal here is to find a model that can predict the class of wine given the 13 measured parameters and find out the major differences among the three different classes. This is a classification problem and here I will describe four models and asses the accuracy of each model. Furthermore, I will use principal component analysis to identify and explore the differences among the three classes.
Multinomial Logistic Regression
As there are three classes of wine, we have to use multinomial logistic regression instead of logistic regression which is used when there are two classes. In order to do that, I use multinom function from the nnet package.
> dim(wine)[1] 178 14> attach(wine)> test=sample(178,45)> library(nnet)> LogReg=multinom(class~.,data=wine[-test,])> summary(LogReg)> Pre=predict(LogReg,wine[test,])> table(Pre,wine[test,]$class)
As we can see from Table 1, there are 5 misclassifications out of 45 observations; so the accuracy of the multinomial logistic regression model is 89%.
We can repeat the above process multiple times to get a more accurate estimate of the performance of Multinomial Logistic Regression model by following these commands:
> Accuracy=rep(0,50)> for (i in 1:50) {+ test=sample(178,45)+ LogReg=multinom(class~.,data=wine[-test,])+ Pre=predict(LogReg,wine[test,])+ Accuracy[i]=mean(Pre==wine[test,]$class)+ }> sum(Accuracy)/50[1] 0.944
Linear Discriminant Analysis (LDA)
LDA is useful when we have more than two classes and when the number of observations is small. LDA is also more stable when distribution of the predictors are normal in each class.
> library(MASS)> lda.fit=lda(class~.,data=wine[-test,])> lda.fit
The last command will generate more details about the model as shown in Table 2.
We then evaluate the performance of the model on test data:
> lda.pred=predict(lda.fit,wine[test,])> table(lda.pred$class,wine[test,]$class)
We can see from Table 3 that LDA has accuracy of 100% in predicting classes of test data.
We can also visualize classification of training data by LDA using below command and result is shown in Figure 1:
> plot(lda.fit)
As there are three classes in data set, only two linear discriminants are needed to classify each observation. Figure 1 shows plot of training data on LD1 and LD2 space and the corresponding class for each data point. LD1 and LD2 values are computed based on coefficients of LDA model.
We can repeat the above process multiple times to get a more accurate estimate of the performance of LDA model by following these commands:
> for (i in 1:50) {+ test=sample(178,45)+ lda.fit=lda(class~.,data=wine[-test,])+ lda.pred=predict(lda.fit,wine[test,])+ Accuracy[i]=mean(lda.pred$class==wine[test,]$class)+ }> sum(Accuracy)/50[1] 0.9844444
Quadratic Discriminant Analysis (QDA)
Another classifier is QDA model which has a similar syntax as LDA in R. We can run the process many times to get a more accurate estimate of the performance of the QDA model as follows:
> qda.fit=qda(class~.,data=wine[-test,])> qda.pred=predict(qda.fit,wine[test,])> table(qda.pred$class,wine[test,]$class)> for (i in 1:50) {+ test=sample(178,45)+ qda.fit=qda(class~.,data=wine[-test,])+ qda.pred=predict(qda.fit,wine[test,])+ Accuracy[i]=mean(qda.pred$class==wine[test,]$class)+ }> sum(Accuracy)/50[1] 0.9866667
K-Nearest Neighbors (KNN)
KNN is a non-parametric approach in which an observation is classified based on the class of its K-nearest neighbors. It’s a useful model when the decision boundary is non-linear but it will not tell us about which predictors are important.
> library(class)> knn.pred=knn(wine[-test,2:14],wine[test,2:14],wine[-test,]$class,k=1)> table(knn.pred,wine[test,]$class)> mean(knn.pred==wine[test,]$class)[1] 0.7777778
We can repeat the above process multiple times to get a more accurate estimate of the performance of KNN model by following these commands:
> for (i in 1:50){+ test=sample(178,45)+ knn.pred=knn(wine[-test,2:14],wine[test,2:14],wine[-test,]$class,k=1)+ Accuracy[i]=mean(knn.pred==wine[test,]$class)+ }> sum(Accuracy)/50[1] 0.7435556
We can repeat the same process for k=2 to 5 and results are shown in the middle column of Table 4.
We can see from the middle column of Table 4 that results of the KNN model are not very impressive. That is because KNN model uses Euclidean distance to measure the distance between two points and if features have different scales it can impact the model. As each of the 13 features have different scales, it is important to normalize data so that all features have the same range of values. We can re-run the KNN model after scaling data as follows:
> for (i in 1:50){+ test=sample(178,45)+ knn.pred=knn(scale(wine[-test,2:14]),scale(wine[test,2:14]),wine[-test,]$class,k=1)+ Accuracy[i]=mean(knn.pred==wine[test,]$class)+ }> sum(Accuracy)/50[1] 0.9382222
Results of KNN model are summarized in Table 4 and we can see that scaling data greatly improve the performance of the model.
The accuracy of different classification models on the wine data set is summarized in Table 5. LDA and QDA have the highest accuracy followed by KNN (k=5) model.
Principal Component Analysis (PCA)
The models that were described above can predict the class of wine based on 13 measured predictors. However, we are also interested to know what the major differences among each of these three classes are and what predictors are important. In order to do that, we can perform PCA which is a useful tool for exploratory data analysis.
> pr.out=prcomp(wine[,-1],scale=TRUE)> pr.out$rotation> biplot(pr.out,scale=0)> plot(pr.out$x[,1:2],col=wine$class)
The first two PC scores and the corresponding loading vectors are shown in Figure 2.
Figure 2 and Figure 3 show that data points are segregated into three different groups which corresponds to the three classes of wine. Class 1 and Class 3 have relatively the same PC2 scores but their PC1 score differs a lot. On the other hand, Class 2 has PC1 scores between that of Class 1 and Class 3 and its PC2 score is lower than the other two classes. We can further examine the difference between each class by looking into PC loading vectors (arrows in Figure 2). For example, the direction of ‘alcalinity of ash’ is towards high values of PC1 and Class 3 data points. Therefore, we can expect that Class 3 have high values of ‘alcalinity of ash’ followed by Class 2 and Class 1. We can do similar investigation for all the other 13 predictors. The results of such investigation are summarized in Tables 6 and 7 which show the major differences among the three classes of wine. These findings are in line with the results of LDA shown in Table 2.
Four classification methods were used to evaluate the accuracy of each model in predicting classes of wine. QDA and LDA have the highest accuracy followed by KNN and multinomial logistic regression. It is important to normalize data before applying KNN model to get accurate classification. Principal component analysis was used to identify the major differences between the three classes of wine.
UCI Machine Learning Repository: Wine Data Set. “ Creative Commons Attribution 4.0 International (CC BY 4.0) license”.
An Introduction to Statistical Learning (statlearning.com)