K-nearest Neighbors Algorithm with Examples in R (Simply Explained knn) | by Kshitiz Sirohi | Towards Data Science

In this post I am going to exampling what k- nearest neighbor algorithm is and how does it help us. But before we move ahead, we aware that my target audience is the one who wants to get intuitive understanding of the concept and not very in-dept understanding, that is why I have avoided being too pedantic about this topic with very less focus on theoretical concept. Let’s get started.
Let’ s say we are given certain number of objects and each object has its own unique attribute associated with it, for example, you have 5 chairs 10 beds and 15 tables and each for each one of them we know the length, breadth and height. Now, if someone give us a new object with new attributes and ask us to predict as to which category does that new object belongs to, that means we are given the dimension and asked to predict if it is a chair, bed or table, then we would have to use knn algorithm to determine that.
Therefore, attributes means the property of each object, and each object can be considered as a category. Our job is to check how closely are the properties of the new object is related to any one of the already known categories. Now, I am going to show you many kinds of scenarios and I will try to explain those as simply possible as I can.
When we are given the attributes, we try and plot them on graph. The graphical representation of those attributes will help us to calculate Euclidian distance between the new value and the ones that we already know we have. By doing so, we can be sure as to which category does the new object is closest to.
Don’t get intimidated by the name, it just simply means the distance between two points in a plane. By simple using this formula you can calculate distance between two points no matter how many attributes or properties you are given like height, breadth, width, weight and so on upto n where n could be the last property of the object you have. The formula is √(x2−x1)2+(y2−y1)2+(z2−z1)2 ...... (n2-n1)2
We have two categories called male and female with their respective heights given in the table below. Then, if a new member shows up and you are asked to determine if it is a male or female given you already know his/her height, then you can plot that height in 1D plane and the check the proximity of that new height with other already drawn heights. Ideally speaking, we would calculate the Euclidean distance on that graph to determine the closest heights to height of new member. We see that if we draw 180cm in height graph, it lies closer to the male heights than it is from female height. That is how we determine it is a male. (a) shows the table (b) shows the representation on graph (c) shows introduction of new member (d) shows the predicted value to be male.
Let’s say we bring up a new attribute called weight that can also describe the characteristic of male and female as shown in the table below. Now we create a 2D plane and follow the same procedure of calculating the distance of new object from the older ones, and the closer it is to either one of the categories will be considered its own category. In the example below, we can see that new member with 169cm of height and 68kg of weight is in proximity with females more than males. Hence, we predict that the new member is a female.
Most often than not, you will be having a lot of attributes associated with your categorical object and they cannot simply be drawn on a 2D or 1D plane on a white board. However, as long as you understand the functionality you are good to go. Let’s say you have 5 attributes viz. sex, height, weight, color, hair_length and voice_pitch then you simply use the above stated Euclidean formula to calculate distance of new upcoming object from the already given objects. You take the square root of (sex1-sex2)2 + (height1- height2)2 + (weight1 — weight2)2 + (color1 — color2)2 + (hair_length1 — hair_length2)2 + (voice_pitch1)2 — (voice_pitch2)2 where 1 is representing already drawn data point and 2 is representing the new data point that you want to determine the category of. Remember, this single formula above will be used with each and every data point available, that means, it will run as many times as there are rows in your dataset.
Before we move on to implementing them in R, be aware of these following notes:
1- The nearest neighbor you want to check will be called defined by value “k”. If k is 5 then you will check 5 closest neighbors in order to determine the category. If majority of neighbor belongs to a certain category from within those five nearest neighbors, then that will be chosen as the category of upcoming object. Shown in the picture below.
2- Different variables have different scaling units, like weight in kg and height in cm. Then how do we suppose to use them in the Euclidean formula? Well, we normalize each one of the variables using the formula (x-min(x))/(min(x) — max(x)) that we will also see in the examples below. Now if you have one variable of 200kg and another with 50kg, after normalization both will be represented by the value between 0 and 1. Just imagine you have created a box that can hold things that are in range of 0 and 1 only, but since you have used normalization formula to converted everything into values between 0 and 1, you are good to use that box. Note: That box is your plot.
3- The knn algorithm works well with the numeric variables, this is not to say that it cannot work with categorical variables, but it’s just if you have mix of both categorical and numeric variables as the predictors then it demands little bit of different approach. But if all predictors are numeric, then knn is best because we are dealing with the distance and for that we need hard numbers.
4- When we split our data into training and testing sets, the data should have already be normalized. That mean we first normalize the data and then split it.
5- The knn algorithm does not works with ordered-factors in R but rather with factors. We will see that in the code below.
6- The k-mean algorithm is different than K- nearest neighbor algorithm. K-mean is used for clustering and is a unsupervised learning algorithm whereas Knn is supervised leaning algorithm that works on classification problems.
df <- data(iris) ##load data head(iris) ## see the studcture##   Sepal.Length Sepal.Width Petal.Length Petal.Width Species ## 1          5.1         3.5          1.4         0.2  setosa ## 2          4.9         3.0          1.4         0.2  setosa ## 3          4.7         3.2          1.3         0.2  setosa ## 4          4.6         3.1          1.5         0.2  setosa ## 5          5.0         3.6          1.4         0.2  setosa ## 6          5.4         3.9          1.7         0.4  setosa##Generate a random number that is 90% of the total number of rows in dataset. ran <- sample(1:nrow(iris), 0.9 * nrow(iris))   ##the normalization function is created nor <-function(x) { (x -min(x))/(max(x)-min(x))   }  ##Run nomalization on first 4 coulumns of dataset because they are the predictors iris_norm <- as.data.frame(lapply(iris[,c(1,2,3,4)], nor))  summary(iris_norm)##   Sepal.Length     Sepal.Width      Petal.Length     Petal.Width      ##  Min.   :0.0000   Min.   :0.0000   Min.   :0.0000   Min. :0.00 ##  1st Qu.:0.2222   1st Qu.:0.3333   1st Qu.:0.1017   1st Qu.:0.08   ##  Median :0.4167   Median :0.4167   Median :0.5678   Median :0.50 ##  Mean   :0.4287   Mean   :0.4406   Mean   :0.4675   Mean   :0.45 ##  3rd Qu.:0.5833   3rd Qu.:0.5417   3rd Qu.:0.6949   3rd Qu.:0.70 ##  Max.   :1.0000   Max.   :1.0000   Max.   :1.0000   Max.   :1.00##extract training setiris_train <- iris_norm[ran,] ##extract testing set iris_test <- iris_norm[-ran,]  ##extract 5th column of train dataset because it will be used as 'cl' argument in knn function. iris_target_category <- iris[ran,5] ##extract 5th column if test dataset to measure the accuracy iris_test_category <- iris[-ran,5]##load the package class library(class) ##run knn function pr <- knn(iris_train,iris_test,cl=iris_target_category,k=13)  ##create confusion matrix tab <- table(pr,iris_test_category)  ##this function divides the correct predictions by total number of predictions that tell us how accurate teh model is.  accuracy <- function(x){sum(diag(x)/(sum(rowSums(x)))) * 100} accuracy(tab)## [1] 80
In the iris dataset that is already available in R, I have run the k-nearest neighbor algorithm that gave me 80% accurate result. First, I normalized the data to convert petal.length, sepal.length, petal.width and sepal.length into a standardized 0-to-1 form so that we can fit them into one box (one graph) and also because our main objective is to predict whether a flower is virginica, Versicolor, or setosa and that is why I excluded the column 5 and stored it into another variable called iris_target_category. Then, I separated the normalized values into training and testing dataset. Imagine it this way, that the values from training dataset are firstly drawn on a graph and after we run knn function with all the necessary arguments, we introduce testing dataset’s values into the graph and calculate Euclidean distance with each and every already stored point in graph. Now, although we know which flower it is in testing dataset, we still predict the values and store them in variable called ‘pr’ so that we can compare predicted values with original testing dataset’s values. This way we understand the accuracy of our model and if we are to get new 50 values in future and we are asked to predict the category of those 50 values, we can do that with this model.
##because diamonds dataset is in ggplot2 package library(ggplot2)##load data data(diamonds)  ##store it as data frame dia <- data.frame(diamonds)  ##create a random number equal 90% of total number of rows ran <- sample(1:nrow(dia),0.9 * nrow(dia))  ##the normalization function is created nor <-function(x) { (x -min(x))/(max(x)-min(x))   }  ##normalization function is created dia_nor <- as.data.frame(lapply(dia[,c(1,5,6,7,8,9,10)], nor))  ##training dataset extracted dia_train <- dia_nor[ran,]  ##test dataset extracted dia_test <- dia_nor[-ran,]##the 2nd column of training dataset because that is what we need to predict about testing dataset ##also convert ordered factor to normal factor dia_target <- as.factor(dia[ran,2])  ##the actual values of 2nd couln of testing dataset to compaire it with values that will be predicted ##also convert ordered factor to normal factor test_target <- as.factor(dia[-ran,2])  ##run knn function library(class) pr <- knn(dia_train,dia_test,cl=dia_target,k=20)  ##create the confucion matrix tb <- table(pr,test_target)  ##check the accuracy accuracy <- function(x){sum(diag(x)/(sum(rowSums(x)))) * 100} accuracy(tb)## [1] 71.09752
In this model I have tried to predict the “cut” variable since it is a categorical variable and we know knn works better for classification problems. I have not taken the 3rd and 4th column into consideration since they were also categorical and would cause ambiguity because other predictive variables are integers. Note that there are ways to run knn on mixture of categorical and integer variables, but we are not discussing it here. The rest of the procedure is same as the iris dataset, and in the end we get the accurate result 71% of the times.
Anonymous. (9 Mar 2015). “R — kNN — k nearest neighbor (part 1)”. Jalayer Academy. https://www.youtube.com/watch?v=GtgJEVxl7DY
Kumar, S. (8 Oct 2017). “kNN(k-Nearest Neighbors) Algorithm in R”. RSTUDIO. https://rstudio-pubs-static.s3.amazonaws.com/316172_a857ca788d1441f8be1bcd1e31f0e875.html
Bhalla, D. (18 Dec 2017). “K Nearest Neighbor: Step by Step Tutorial”. RBLOGGERS. https://www.r-bloggers.com/k-nearest-neighbor-step-by-step-tutorial/