k-NN on Iris Dataset. k-Nearest Neighbor(k-NN) is an... | by Ishita Kapur | Towards Data Science

The class of an unknown instance is computed using the following steps:
The distance between the unknown instance and all other training instances is computed.The k nearest neighbors are identified.The class labels of the k nearest neighbors are used to determine the class label of the unknown instance by using techniques like majority voting.
The distance between the unknown instance and all other training instances is computed.
The k nearest neighbors are identified.
The class labels of the k nearest neighbors are used to determine the class label of the unknown instance by using techniques like majority voting.
For example, In the image above the classifier can classify the unknown instance to belong to class B if the value of k is 3 and to class A if the value of k is 7.
This post focuses on hyperparameter tuning for kNN using the Iris dataset. The optimal hyperparameters are then used to classify the test set instances and compute the final accuracy of the model. The implementation has been done from scratch with no dependencies on existing python data science libraries.
The hyperparameters tuned are:
Distance Metrics: Euclidean, Normalized Euclidean and Cosine Similarityk-values: 1, 3, 5, and 7
Distance Metrics: Euclidean, Normalized Euclidean and Cosine Similarity
k-values: 1, 3, 5, and 7
Euclidean DistanceEuclidean Distance between two points p and q in the Euclidean space is computed as follows:
Normalized Euclidean DistanceNormalized Euclidean distance is the euclidean distance between points after the points have been normalized.
Cosine SimilarityCosine Similarity is the similarity measure between two non-zero vectors. Cosine Similarity between two vectors A and B is computed as follows:
Now, let’s get into the implementation of the classifier which is done in Python. This section has been divided into different steps for the ease of understanding.
Step 1.
Load the libraries which are required.
import pandas as pdimport numpy as npimport operatorimport matplotlib.pyplot as plt
Step 2.
Load the Iris dataset into the Jupyter Notebook. The dataset is in the csv format and can easily be read into a dataframe using the Pandas library. The dataset has four attributes sepal length, sepal width, petal length, and petal width and also a class label for each instance.
data = pd.read_csv('iris.data', header=None, names=['sepal_length', 'sepal_width', 'petal_length', 'petal_width', 'class'])
The dataframe can be displayed as follows:
print(data)
Step 3.
Divide the dataset into development set and test set. It can be divided by randomizing the indices and then splitting the dataframe according to the indices.
indices = np.random.permutation(data.shape[0])div = int(0.75 * len(indices))development_id, test_id = indices[:div], indices[div:]development_set, test_set = data.loc[development_id,:], data.loc[test_id,:]print("Development Set:\n", development_set, "\n\nTest Set:\n", test_set)
Compute the mean and standard deviation of the development and test sets, to compute the normalized euclidean distance.
mean_development_set = development_set.mean()mean_test_set = test_set.mean()std_development_set = development_set.std()std_test_set = test_set.std()
Retrieve the ‘class’ column from the development and test sets and store it in separate lists.
test_class = list(test_set.iloc[:,-1])dev_class = list(development_set.iloc[:,-1])
Step 4.
Define functions to compute the value of the distance metrics: Euclidean, Normalized Euclidean and Cosine Similarity1
def euclideanDistance(data_1, data_2, data_len):    dist = 0    for i in range(data_len):        dist = dist + np.square(data_1[i] - data_2[i])    return np.sqrt(dist)def normalizedEuclideanDistance(data_1, data_2, data_len, data_mean, data_std):    n_dist = 0    for i in range(data_len):        n_dist = n_dist + (np.square(((data_1[i] - data_mean[i])/data_std[i]) - ((data_2[i] - data_mean[i])/data_std[i])))    return np.sqrt(n_dist)def cosineSimilarity(data_1, data_2):    dot = np.dot(data_1, data_2[:-1])    norm_data_1 = np.linalg.norm(data_1)    norm_data_2 = np.linalg.norm(data_2[:-1])    cos = dot / (norm_data_1 * norm_data_2)    return (1-cos)
Define the function to return k-Nearest Neighbors2
def knn(dataset, testInstance, k, dist_method, dataset_mean, dataset_std):     distances = {}    length = testInstance.shape[1]    if dist_method == 'euclidean':        for x in range(len(dataset)):            dist_up = euclideanDistance(testInstance, dataset.iloc[x], length)            distances[x] = dist_up[0]    elif dist_method == 'normalized_euclidean':        for x in range(len(dataset)):            dist_up = normalizedEuclideanDistance(testInstance, dataset.iloc[x], length, dataset_mean, dataset_std)            distances[x] = dist_up[0]    elif dist_method == 'cosine':        for x in range(len(dataset)):            dist_up = cosineSimilarity(testInstance, dataset.iloc[x])            distances[x] = dist_up[0]    # Sort values based on distance    sort_distances = sorted(distances.items(), key=operator.itemgetter(1))    neighbors = []    # Extracting nearest k neighbors    for x in range(k):        neighbors.append(sort_distances[x][0])    # Initializing counts for 'class' labels counts as 0    counts = {"Iris-setosa" : 0, "Iris-versicolor" : 0, "Iris-virginica" : 0}    # Computing the most frequent class    for x in range(len(neighbors)):        response = dataset.iloc[neighbors[x]][-1]         if response in counts:            counts[response] += 1        else:            counts[response] = 1    # Sorting the class in reverse order to get the most frequest class    sort_counts = sorted(counts.items(), key=operator.itemgetter(1), reverse=True)    return(sort_counts[0][0])
Step 5.
Using the development dataset, iterate over all of the development data instances and compute the class for each k value and each distance metric.
# Creating a list of list of all columns except 'class' by iterating through the development setrow_list = []for index, rows in development_set.iterrows():    my_list =[rows.sepal_length, rows.sepal_width, rows.petal_length, rows.petal_width]           row_list.append([my_list])# k values for the number of neighbors that need to be consideredk_n = [1, 3, 5, 7]# Distance metricsdistance_methods = ['euclidean', 'normalized_euclidean', 'cosine']# Performing kNN on the development set by iterating all of the development set data points and for each k and each distance metricobs_k = {}for dist_method in distance_methods:    development_set_obs_k = {}    for k in k_n:        development_set_obs = []        for i in range(len(row_list)):            development_set_obs.append(knn(development_set, pd.DataFrame(row_list[i]), k, dist_method, mean_development_set, std_development_set))        development_set_obs_k[k] = development_set_obs    # Nested Dictionary containing the observed class for each k and each distance metric (obs_k of the form obs_k[dist_method][k])    obs_k[dist_method] = development_set_obs_k#print(obs_k)
Compute the accuracy for the development set
# Calculating the accuracy of the development set by comparing it with the development set 'class' list created earlieraccuracy = {}for key in obs_k.keys():    accuracy[key] = {}    for k_value in obs_k[key].keys():        #print('k = ', key)        count = 0        for i,j in zip(dev_class, obs_k[key][k_value]):            if i == j:                count = count + 1            else:                pass        accuracy[key][k_value] = count/(len(dev_class))# Storing the accuracy for each k and each distance metric into a dataframedf_res = pd.DataFrame({'k': k_n})for key in accuracy.keys():    value = list(accuracy[key].values())    df_res[key] = valueprint(df_res)
Plot a bar chart to compare the performance of hyperparameters.
# Plotting a Bar Chart for accuracydraw = df_res.plot(x='k', y=['euclidean', 'normalized_euclidean', 'cosine'], kind="bar", colormap='YlGnBu')draw.set(ylabel='Accuracy')
Note: Ignoring k=1 if the value of accuracy for k=1 is 100%, since this mostly implies overfitting. Replace it with the value of numpy.nan.
df_res.loc[df_res['k'] == 1.0, ['euclidean', 'normalized_euclidean', 'cosine']] = np.nan
Step 6.
Find the optimal hyperparameters.
# In case the accuracy is the same for different k and different distance metric selecting the first of all the samecolumn_val = [c for c in df_res.columns if not c.startswith('k')]col_max = df_res[column_val].max().idxmax(1)best_dist_method = col_maxrow_max = df_res[col_max].argmax()best_k = int(df_res.iloc[row_max]['k'])if df_res.isnull().values.any():    print('\n\n\nBest k value is\033[1m', best_k, '\033[0mand best distance metric is\033[1m', best_dist_method, '\033[0m. Ignoring k=1 if the value of accuracy for k=1 is 100%, since this mostly implies overfitting')else:    print('\n\n\nBest k value is\033[1m', best_k, '\033[0mand best distance metric is\033[1m', best_dist_method, '\033[0m.')
Step 7.
Use the test set and the optimal hyperparameters to compute the final accuracy.
# Creating a list of list of all columns except 'class' by iterating through the development setrow_list_test = []for index, rows in test_set.iterrows():     my_list =[rows.sepal_length, rows.sepal_width, rows.petal_length, rows.petal_width]           row_list_test.append([my_list])test_set_obs = []for i in range(len(row_list_test)):    test_set_obs.append(knn(test_set, pd.DataFrame(row_list_test[i]), best_k, best_dist_method, mean_test_set, std_test_set))#print(test_set_obs)count = 0for i,j in zip(test_class, test_set_obs):    if i == j:        count = count + 1    else:        passaccuracy_test = count/(len(test_class))print('Final Accuracy of the Test dataset is ', accuracy_test)
A final accuracy of about 97% is achieved using Cosine Similarity as the distance metric and the k-value of 5. The model gave really good results.
The complete code can be found on the GitHub repository.
[1]: Cosine Similarityhttps://stackoverflow.com/questions/18424228/cosine-similarity-between-2-number-lists
[2]: As a referencehttps://machinelearningmastery.com/tutorial-to-implement-k-nearest-neighbors-in-python-from-scratch/