KNN (K-Nearest Neighbors) #1. How it works? | by Italo José | Towards Data Science

KNN (K — Nearest Neighbors) is one of many (supervised learning) algorithms used in data mining and machine learning, it’s a classifier algorithm where the learning is based “how similar” is a data (a vector) from other .
The KNN is pretty simple, imagine that you have a data about colored balls:
Purple balls;
Yellow balls;
And a ball that you don’t know if it’s purple or yellow, but you has all the data about this color(except the color label).
So, how are you going to know the ball’s color? imagine you like a machine that you only have the ball’s characteristics(data), but doesn’t the final label. Hou do you will to know the ball’s color(final label/your class)?
Obs: Let’s suppose that data with number 1(and label R) are referring to the purple balls and the data with number 2 (and label A) are referring to the yellow balls, this’s just to make the explanation easier, in future articles we will work with real data.
Each line refers to a ball and each column refers to a ball’s characteristic, in the last column we have the class (color) of each of the balls:
R -> purple;
A -> yellow
We have 5 balls there ( 5 lines), each one with yours classification, you can try to discover the new ball’s color (in the case the class) of N ways, one of these N ways is to comparing this new ball’s characteristics with all the others, and see what it looks like most, if the data(characteristics) of this new ball (that you doesn’t know the correct class) is similar to the data of the yellow balls, then the color of the new ball is yellow, if the data in the new ball is more similar to the data of the purple then yellow, then the color of the new ball is purple, it looks so simple, and that is almost what the knn does, but in a most sophisticated way.
In the case of KNN, indeed it doesn’t “compare” the new (unclassified) data with all other, actually he performs a mathematical calculation to measure the distance between the data to makes the classification, (that’s almoooooost the same thing). This calculation that I am talking about, can be anyone other calculation that measures the distance between two points, for example: Euclidian, Manhattan, Minkowski, Weighted ...
The KNN’s steps are:
1 — Receive an unclassified data;
2 — Measure the distance (Euclidian, Manhattan, Minkowski or Weighted) from the new data to all others data that is already classified;
3 — Gets the K(K is a parameter that you difine) smaller distances;
4 — Check the list of classes had the shortest distance and count the amount of each class that appears;
5 — Takes as correct class the class that appeared the most times;
6 —Classifies the new data with the class that you took in step 5;
Below we have an image with all the process that we talk about in this article, you have an unclassified data (in red) and all your other data already classified (yellow and purple), each one with your class (A or B). So you calculate the distances of your new data with all the others to know which ones have the smallest distance, so you get 3 (or 6) of the closest data and check which class that appears most , in the case of the image below, the closest data to the new data are those that are inside the first circle (inside circle), and inside this circle there are 3 other data (already classified with yellow), we will check which is the predominant class in there, look, it’s purple, because there are 2 purple balls and only 1 yellow, sooooooo this new data that before was not classified, now it will be classified as purple.
To calculate the distance between two points (your new sample and all the data you have in your dataset) is very simple, as said before, there are several ways to get this value, in this article we will use the Euclidean distance.
The Euclidean distance’s formule is like the image below:
Wait, don’t worry, it isn’t so much complicated as it sounds, after reading this article and continuing to find it complex, when we get into the code you will see just how simple it is.
Using this formula that you will check the distance between 1 point (your unclassified sample) and 1 other point in your dataset (1 other already classified data), one by one in all your dataset, the smaller the result of this calculation is the most similar between these two data. And yes, you will run this calculation so much times, until you have runned it in all the data, when you do this, you will have an array (let’s use an array as an example) with the result) of your unclassified data to another data already classified.
Ok, let’s do it! It’s simple but read it slowly, basically you’ll:
Get each characteristic from your dataset;
Subtract each one, example, (line 1, column 5) — (line1,column5) = X ...(line 1, column 13) — (line1,column13) = Z;
After get the subtract of all columns, you will get all the results and sum it X+Y +Z... ;
So you wil get the sum’s square root ;
Let’s use the example from the previous worksheet but now with 1 unclassified data, thas’s the information we want to discover.
We have 5 data (lines) in this example, each sample (data/line) have your attributes (characteristics), let’s IMAGINE that all these are images, each line would be an image and each column would be a image’s pixel.
So let’s start by that confuse explain above,.
Let’s take the first line, which is the data that we want to classify and let’s measure the Euclidean distance to line 2
1 — Subtraction
Let’s subtract each attribute (column) from row 1 with the attributes from row 2, example:
(1–2) = -1
2 — Exponentiation:
After you had subtract column 1 from row 1, with column 1 from row 2, we will get the squared root, so the result numbers are aways positive, example:
(1–2)2 = (-1)2(-1)2 = 1
3 — Sum
After you have done step 2, for all the row 1's columns and row 2’s columns, we will sum all these results, let’s make an example using the spreadsheet’s columns’s image and we will have the following result:
(1–2)2 + (1–2)2 + (1–2)2 + (1–2)2 + (1–2)2 + (1–2)2 + (1–2)2 + (1–2)2 = 8
Note that we have 8 attribute columns, both in row 1 and row 2, and we performed step 2 for each dataset’s attribute, so our final result was 8, but why 8? Because each time we run step 2, the result gave 1, by a “coincidence” we have the same data in all columns and the result of (1–2) 2 is equal to 1, I used those values to facilitate the math here, but no,this attributes does’nt need always be the same number, later we will see this better in implementing a code with this algorithm.
4 —Square root:
After performed step 3, we will get our subtractions’s sum’s square root . In step 3, the result was 8, so let’s take the square root of number 8:
√8 = 2,83 ou 8^(1⁄2) = 2,83
Great! Now you have the Euclidean distance from line 1 to line 2, look, it was not so difficult, you could do it in a simple paper!
Now, you only need to make these for all dataset’s lines, from line 1 to all other lines, when you do this, you will have the Euclidean distance from line 1 to all other lines, then you will sort it to get the “k”(e.g. k = 3) smallest distances, so you will check which is the class that most appears, the class that appears the most times will be the class that you will use to classify the line 1 (which was not classified before).
Simple right? In next articles we will start to implement this algorithm in GO! But don’t worry, Go’s syntax is so simple that you’ll be following the tutorial and apply it in your language.