KNN Algorithm: When? Why? How?. KNN: K Nearest Neighbour is one of the... | by Aditya Kumar | Towards Data Science

KNN: K Nearest Neighbor is one of the fundamental algorithms in machine learning. Machine learning models use a set of input values to predict output values. KNN is one of the simplest forms of machine learning algorithms mostly used for classification. It classifies the data point on how its neighbor is classified.
KNN classifies the new data points based on the similarity measure of the earlier stored data points. For example, if we have a dataset of tomatoes and bananas. KNN will store similar measures like shape and color. When a new object comes it will check its similarity with the color (red or yellow) and shape.
K in KNN represents the number of the nearest neighbors we used to classify new data points.
How do I choose K?
In real-life problems where we have many points the question arises is how to select the value of K?
Choosing the right value of K is called parameter tuning and it’s necessary for better results. By choosing the value of K we square root the total number of data points available in the dataset.
a. K = sqrt (total number of data points).
b. Odd value of K is always selected to avoid confusion between 2 classes.
When is KNN?
a. We have properly labeled data. For example, if we are predicting someone is having diabetes or not the final label can be 1 or 0. It cannot be NaN or -1.
b. Data is noise-free. For the diabetes data set we cannot have a Glucose level as 0 or 10000. It’s practically impossible.
c. Small dataset.
How does KNN work?
We usually use Euclidean distance to calculate the nearest neighbor. If we have two points (x, y) and (a, b). The formula for Euclidean distance (d) will be
d = sqrt((x-a)2+(y-b)2)
We try to get the smallest Euclidean distance and based on the number of smaller distances we perform our calculation.
Let’s try KNN on one database to see how it works. The data can be extracted from https://github.com/adityakumar529/Coursera_Capstone/blob/master/diabetes.csv.
import pandas as pdimport numpy as npfrom sklearn.model_selection import train_test_splitfrom sklearn.preprocessing import StandardScalerfrom sklearn.neighbors import KNeighborsClassifierfrom sklearn.metrics import confusion_matrixfrom sklearn.metrics import accuracy_scorefrom sklearn.metrics import f1_score
pd and np are for pandas and NumPy library. The last 3 lines(confusion_matrix,accuracy_score, and f1_score) are for checking the accuracy of the model. train_test_split is to split and train the data. KNeighborsClassifier is for K nearest neighbor. Standardization of datasets is a common requirement for many machine learning estimators: they might behave badly if the individual features do not more or less look like standard normally distributed data.
data = pd.read_csv("../input/diabetes.csv")data.head()
We have read the CSV file through pd.read_csv. And through the head() we can see the top 5 rows. There are some factors where the values cannot be zero. Glucose values, for example, cannot be 0 for a human. Similarly, BloodPressure, SkinThickness, Insulin, and BMI cannot be zero for a human.
non_zero = ['Glucose','BloodPressure','SkinThickness','Insulin','BMI']for coloumn in non_zero:    data[coloumn] = data[coloumn].replace(0,np.NaN)    mean = int(data[coloumn].mean(skipna = True))    data[coloumn] = data[coloumn].replace(np.NaN, mean)    print(data[coloumn])0      148.01       85.02      183.03       89.04      137.0       ...  763    101.0764    122.0765    121.0766    126.0767     93.0Name: Glucose, Length: 768, dtype: float640      72.01      66.02      64.03      66.04      40.0       ... 763    76.0764    70.0765    72.0766    60.0767    70.0Name: BloodPressure, Length: 768, dtype: float640      35.01      29.02      29.03      23.04      35.0       ... 763    48.0764    27.0765    23.0766    29.0767    31.0Name: SkinThickness, Length: 768, dtype: float640      155.01      155.02      155.03       94.04      168.0       ...  763    180.0764    155.0765    112.0766    155.0767    155.0Name: Insulin, Length: 768, dtype: float640      33.61      26.62      23.33      28.14      43.1       ... 763    32.9764    36.8765    26.2766    30.1767    30.4Name: BMI, Length: 768, dtype: float64
We have created non_zero which has all the column which are needed for predicting the outcome value. We need to make sure these columns do not have any value related to zero or NaN value. If we have 0 then we will replace it with NaN. And then replacing NaN with the mean value of the column. Let’s plot details of the diabetes data.
import seaborn as snsp=sns.pairplot(data, hue = 'Outcome')
We have defined non_zero with the column where the values cannot be zero. In each column we will first check if we have 0 values. Then we replace it with NaN. Later we are creating a meaning of the column and replacing the earlier with mean.
Since we have the data ready. It’s time to train and test the data.
X =data.iloc[:,0:8]y =data.iloc[:,8]X_train,X_test,y_train, y_test = train_test_split(X, y, test_size=0.2,random_state=0, stratify=y)
For data X we are taking all the rows of columns ranging from 0 to 7. Similarly, for y we are taking all the rows for the 8th column.
We have train_test_split which we had imported during the start of the program, and we have defined test size as 0.2 which implies out of all the data 20% will be kept aside to test the data at a later stage.
#feature Scalingsc_X = StandardScaler()X_train = sc_X.fit_transform(X_train)X_test = sc_X.transform(X_test)
StandardScaler performs the task of Standardization. Usually a dataset contains variables that are different in scale. For example, the dataset will contain an insulin column with values on scale 20–70 and Glucose column with values on scale 80–200. As these two columns are different in scale, they are standardized to have a common scale while building a machine learning model.
import mathmath.sqrt(len(y_test))
Out:
12.409673645990857
We have taken this value to get the value of K. We need an odd value of K, so we will make it 12–1 or 12+1.
classifier = KNeighborsClassifier(n_neighbors=13,p=2,metric='euclidean')classifier.fit(X_train,y_train)
Out:
KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='euclidean',                     metric_params=None, n_jobs=None, n_neighbors=13, p=2,                     weights='uniform')
Let’s predict our data using classifier predict.
y_pred =  classifier.predict(X_test)y_pred
Out:
array([0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1,       1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0, 0, 0, 1, 0,       1, 1, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0,       0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 1, 0, 0,       0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0,       0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 1,       0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0])
We have an array of data, but we need to evaluate our model to check the accuracy. Let’s start with a confusion matrix.
cm= confusion_matrix(y_test,y_pred)cm
Out:
array([[86, 14],       [24, 30]])
We have a confusion matrix where the diagonal with 86 and 30 shows the correct value and 14, 24 shows the prediction that we missed.
We will check the f1 score.
print(f1_score(y_test,y_pred))
Out
0.6122448979591836print(accuracy_score(y_test,y_pred))
Out:
0.7532467532467533
We have f1 score as 0.61 and an accuracy score of .75
Let’s plot the graph for the actual data and our predicted value.
import matplotlib.pyplot as pltplt.figure(figsize=(5, 7))ax = sns.distplot(data['Outcome'], hist=False, color="r", label="Actual Value")sns.distplot(y_pred, hist=False, color="b", label="Predicted Values", ax=ax)plt.title('Actual vs Precited value for outcome')plt.show()plt.close()
The actual code can be checked out on: