Solving A Simple Classification Problem with Python — Fruits Lovers’ Edition | by Susan Li | Towards Data Science

In this post, we’ll implement several machine learning algorithms in Python using Scikit-learn, the most popular machine learning tool for Python. Using a simple dataset for the task of training a classifier to distinguish between different types of fruits.
The purpose of this post is to identify the machine learning algorithm that is best-suited for the problem at hand; thus, we want to compare different algorithms, selecting the best-performing one. Let’s get started!
The fruits dataset was created by Dr. Iain Murray from University of Edinburgh. He bought a few dozen oranges, lemons and apples of different varieties, and recorded their measurements in a table. And then the professors at University of Michigan formatted the fruits data slightly and it can be downloaded from here.
Let’s have a look the first a few rows of the data.
%matplotlib inlineimport pandas as pdimport matplotlib.pyplot as pltfruits = pd.read_table('fruit_data_with_colors.txt')fruits.head()
Each row of the dataset represents one piece of the fruit as represented by several features that are in the table’s columns.
We have 59 pieces of fruits and 7 features in the dataset:
print(fruits.shape)
(59, 7)
We have four types of fruits in the dataset:
print(fruits['fruit_name'].unique())
[‘apple’ ‘mandarin’ ‘orange’ ‘lemon’]
The data is pretty balanced except mandarin. We will just have to go with it.
print(fruits.groupby('fruit_name').size())
import seaborn as snssns.countplot(fruits['fruit_name'],label="Count")plt.show()
Box plot for each numeric variable will give us a clearer idea of the distribution of the input variables:
fruits.drop('fruit_label', axis=1).plot(kind='box', subplots=True, layout=(2,2), sharex=False, sharey=False, figsize=(9,9),                                         title='Box Plot for each input variable')plt.savefig('fruits_box')plt.show()
It looks like perhaps color score has a near Gaussian distribution.
import pylab as plfruits.drop('fruit_label' ,axis=1).hist(bins=30, figsize=(9,9))pl.suptitle("Histogram for each numeric input variable")plt.savefig('fruits_hist')plt.show()
Some pairs of attributes are correlated (mass and width). This suggests a high correlation and a predictable relationship.
from pandas.tools.plotting import scatter_matrixfrom matplotlib import cmfeature_names = ['mass', 'width', 'height', 'color_score']X = fruits[feature_names]y = fruits['fruit_label']cmap = cm.get_cmap('gnuplot')scatter = pd.scatter_matrix(X, c = y, marker = 'o', s=40, hist_kwds={'bins':15}, figsize=(9,9), cmap = cmap)plt.suptitle('Scatter-matrix for each input variable')plt.savefig('fruits_scatter_matrix')
We can see that the numerical values do not have the same scale. We will need to apply scaling to the test set that we computed for the training set.
from sklearn.model_selection import train_test_splitX_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)from sklearn.preprocessing import MinMaxScalerscaler = MinMaxScaler()X_train = scaler.fit_transform(X_train)X_test = scaler.transform(X_test)
from sklearn.linear_model import LogisticRegressionlogreg = LogisticRegression()logreg.fit(X_train, y_train)print('Accuracy of Logistic regression classifier on training set: {:.2f}'     .format(logreg.score(X_train, y_train)))print('Accuracy of Logistic regression classifier on test set: {:.2f}'     .format(logreg.score(X_test, y_test)))
Accuracy of Logistic regression classifier on training set: 0.70Accuracy of Logistic regression classifier on test set: 0.40
from sklearn.tree import DecisionTreeClassifierclf = DecisionTreeClassifier().fit(X_train, y_train)print('Accuracy of Decision Tree classifier on training set: {:.2f}'     .format(clf.score(X_train, y_train)))print('Accuracy of Decision Tree classifier on test set: {:.2f}'     .format(clf.score(X_test, y_test)))
Accuracy of Decision Tree classifier on training set: 1.00Accuracy of Decision Tree classifier on test set: 0.73
from sklearn.neighbors import KNeighborsClassifierknn = KNeighborsClassifier()knn.fit(X_train, y_train)print('Accuracy of K-NN classifier on training set: {:.2f}'     .format(knn.score(X_train, y_train)))print('Accuracy of K-NN classifier on test set: {:.2f}'     .format(knn.score(X_test, y_test)))
Accuracy of K-NN classifier on training set: 0.95Accuracy of K-NN classifier on test set: 1.00
from sklearn.discriminant_analysis import LinearDiscriminantAnalysislda = LinearDiscriminantAnalysis()lda.fit(X_train, y_train)print('Accuracy of LDA classifier on training set: {:.2f}'     .format(lda.score(X_train, y_train)))print('Accuracy of LDA classifier on test set: {:.2f}'     .format(lda.score(X_test, y_test)))
Accuracy of LDA classifier on training set: 0.86Accuracy of LDA classifier on test set: 0.67
from sklearn.naive_bayes import GaussianNBgnb = GaussianNB()gnb.fit(X_train, y_train)print('Accuracy of GNB classifier on training set: {:.2f}'     .format(gnb.score(X_train, y_train)))print('Accuracy of GNB classifier on test set: {:.2f}'     .format(gnb.score(X_test, y_test)))
Accuracy of GNB classifier on training set: 0.86Accuracy of GNB classifier on test set: 0.67
from sklearn.svm import SVCsvm = SVC()svm.fit(X_train, y_train)print('Accuracy of SVM classifier on training set: {:.2f}'     .format(svm.score(X_train, y_train)))print('Accuracy of SVM classifier on test set: {:.2f}'     .format(svm.score(X_test, y_test)))
Accuracy of SVM classifier on training set: 0.61Accuracy of SVM classifier on test set: 0.33
The KNN algorithm was the most accurate model that we tried. The confusion matrix provides an indication of no error made on the test set. However, the test set was very small.
from sklearn.metrics import classification_reportfrom sklearn.metrics import confusion_matrixpred = knn.predict(X_test)print(confusion_matrix(y_test, pred))print(classification_report(y_test, pred))
import matplotlib.cm as cmfrom matplotlib.colors import ListedColormap, BoundaryNormimport matplotlib.patches as mpatchesimport matplotlib.patches as mpatchesX = fruits[['mass', 'width', 'height', 'color_score']]y = fruits['fruit_label']X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)def plot_fruit_knn(X, y, n_neighbors, weights):    X_mat = X[['height', 'width']].as_matrix()    y_mat = y.as_matrix()# Create color maps    cmap_light = ListedColormap(['#FFAAAA', '#AAFFAA', '#AAAAFF','#AFAFAF'])    cmap_bold  = ListedColormap(['#FF0000', '#00FF00', '#0000FF','#AFAFAF'])clf = neighbors.KNeighborsClassifier(n_neighbors, weights=weights)    clf.fit(X_mat, y_mat)# Plot the decision boundary by assigning a color in the color map    # to each mesh point.        mesh_step_size = .01  # step size in the mesh    plot_symbol_size = 50        x_min, x_max = X_mat[:, 0].min() - 1, X_mat[:, 0].max() + 1    y_min, y_max = X_mat[:, 1].min() - 1, X_mat[:, 1].max() + 1    xx, yy = np.meshgrid(np.arange(x_min, x_max, mesh_step_size),                         np.arange(y_min, y_max, mesh_step_size))    Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])# Put the result into a color plot    Z = Z.reshape(xx.shape)    plt.figure()    plt.pcolormesh(xx, yy, Z, cmap=cmap_light)# Plot training points    plt.scatter(X_mat[:, 0], X_mat[:, 1], s=plot_symbol_size, c=y, cmap=cmap_bold, edgecolor = 'black')    plt.xlim(xx.min(), xx.max())    plt.ylim(yy.min(), yy.max())patch0 = mpatches.Patch(color='#FF0000', label='apple')    patch1 = mpatches.Patch(color='#00FF00', label='mandarin')    patch2 = mpatches.Patch(color='#0000FF', label='orange')    patch3 = mpatches.Patch(color='#AFAFAF', label='lemon')    plt.legend(handles=[patch0, patch1, patch2, patch3])plt.xlabel('height (cm)')plt.ylabel('width (cm)')plt.title("4-Class classification (k = %i, weights = '%s')"           % (n_neighbors, weights))    plt.show()plot_fruit_knn(X_train, y_train, 5, 'uniform')
k_range = range(1, 20)scores = []for k in k_range:    knn = KNeighborsClassifier(n_neighbors = k)    knn.fit(X_train, y_train)    scores.append(knn.score(X_test, y_test))plt.figure()plt.xlabel('k')plt.ylabel('accuracy')plt.scatter(k_range, scores)plt.xticks([0,5,10,15,20])
For this particular dateset, we obtain the highest accuracy when k=5.
In this post, we focused on the prediction accuracy. Our objective is to learn a model that has a good generalization performance. Such a model maximizes the prediction accuracy. We identified the machine learning algorithm that is best-suited for the problem at hand (i.e. fruit types classification); therefore, we compared different algorithms and selected the best-performing one.
Source code that created this post can be found here. I would be pleased to receive feedback or questions on any of the above.