K-Nearest Neighbors
In my previous article, I went over the Bayes’ classifier, which is proved to be optimal. But the problem here is that statistical approach…
Category: Machine Learning
Author: Suhyun Kim

K-Nearest Neighbors
In my previous article, I went over the Bayes’ classifier, which is proved to be optimal. But the problem here is that statistical approach requires a lot of extra work. So one option is to consider another classifier, which does not have parameters.
Parametrized classifiers like MLE classifier or Bayes’ classifier require the decision on probability models, which results in extra work in the beginning; thus, it’s error-prone. But with that underlying assumption of the probability model, one can get so much information about the data such as if a given point is an outlier or not.

Here, we will take a look at K-Nearest Neighbors classifier, which is an unparametrized classifier.
http://www.cs.columbia.edu/~verma/classes/ml/lec/lec1_intro_mle_bayes_naive_evaluation.pdf
In the figure above, we are trying to figure out if the white dot belongs to the red or blue. We can simply look at its nearest neighbors and predict what classification it belongs to. That is K-nearest neighbors classifier. K differs depending on how many data you want to look at before making a decision.
But the problem is how are we going to calculate the distance between data points?
Euclidean distance:
http://www.cs.columbia.edu/~verma/classes/ml/lec/lec1_intro_mle_bayes_naive_evaluation.pdf
2. Other normed distances:
http://www.cs.columbia.edu/~verma/classes/ml/lec/lec1_intro_mle_bayes_naive_evaluation.pdf
p = 2: euclidean distance
p = 1: manhattan distance (no diagonal distance is calculated; only vertical and horizontal distance is calculated)
p = ∞: Only maximum distance is of interest
p = 0: count the number of non-zero distance e.g. if there are two dimensions and if there is a difference in dimension 1 and there is no difference in dimension 2, the count is 1.
Instead of calculating distance, can we calculate something else?
Yes, we can calculate similarities between data.
Typically, it is some sort of a relation from the distance calculation like this:
http://www.cs.columbia.edu/~verma/classes/ml/lec/lec1_intro_mle_bayes_naive_evaluation.pdf
But instead of that, we can use the angles between the data using cosine similarity:
http://www.cs.columbia.edu/~verma/classes/ml/lec/lec1_intro_mle_bayes_naive_evaluation.pdf
Then, there is no distance calculated between data points at all. This is what’s actually used for recommendation for similar videos in Netflix.
K-NN’s Optimality
For fixed k, as and n → ∞, k-NN classifier error converges to no more than twice Bayes classifier error.
If k → ∞, k/n → 0, and n → ∞, k-NN classifier converges to Bayes classifier. Therefore, if Bayes’ is optimal, k-NN is optimal.
But there is a drawback of K-NN that it is sensitive to noise in data, which would result in noise in my prediction as well. This problem can be solved by taking majority of k-nearest neighbors. Also, there is no best of way to measure approximate closeness. More importantly, all the training data need to be kept in order to give relation between the given data and training data.
Another drawback is it takes time to find the k-nearest neighbors. The running time it takes is O(nd) where n is the number of training data and d is the dimension. In order to make this problem simpler, how about we turn this into a decision question: What are the k-nearest neighbors (optimization problem) -> Does an element x exist in the nearest approximation (decision problem). In order to answer the decision problem, a naive approach would result in O(n).
In order to do it quickly, a k-d tree can be implemented. The k-d tree is “a binary tree in which every node is a k-dimensional point”. Areas are partitioned and represented in a tree form to implement the binary search. If the points are sorted, the search time significantly decreases from O(n) to O(log n).
http://www.cs.columbia.edu/~verma/classes/ml/lec/lec1_intro_mle_bayes_naive_evaluation.pdf
Drawback of K-d tree
It doesn’t give the most accurate nearest neighbor.
http://www.cs.columbia.edu/~verma/classes/ml/lec/lec1_intro_mle_bayes_naive_evaluation.pdf
For example, if I am looking for the nearest neighbor of the red dot from the above diagram, instead of giving me the blue dot, it will give the green dot on the left hand side of the red dot, because the binary search implemented with k-d tree will result in the T1,1 area and will look for the nearest neighbor in that subregion. However, this drawback doesn’t stop people from using k-d tree, because O(log n) running time is more beneficial over O(n) and preciseness can be sacrificed.
Another way to find the neighbor quickly: Binary search with median
The diagram below is a 2-dimensional example. Instead of looking for the middle partitioned area, we can look for the medean data point instead.
http://www.cs.columbia.edu/~verma/classes/ml/lec/lec1_intro_mle_bayes_naive_evaluation.pdf
On top of that, in order to accommodate to the 2-dimensional aspect, we can start with the dimension that has more data points than the other dimension. If the size of the data point matrix is m by n, and m > n, we can do binary partition in the m dimension. Then the next time, if m > n, we do the partition again in the m dimension. If m < n, then we move onto the partition in the n dimension
Metric learning
How to make measurement robust depending on the parameters? We can give weights to each parameter and add the weights when we calculate the distance. Then now the question becomes how can we find the optimal weights?
http://www.cs.columbia.edu/~verma/classes/ml/lec/lec1_intro_mle_bayes_naive_evaluation.pdf
We can define a cost function for the algorithm to learn the weight on its own.
http://www.cs.columbia.edu/~verma/classes/ml/lec/lec1_intro_mle_bayes_naive_evaluation.pdf
But the problem is now we need to keep all the data around. So motivated by k-nearest neighbors implemented with k-d tree, there is another classifier, which is called decision tree.
Decision Tree
Decision trees in general are highly used in the industry. For instance, Amazon uses decision trees for their fraud order detection.
http://www.cs.columbia.edu/~verma/classes/ml/lec/lec1_intro_mle_bayes_naive_evaluation.pdf
With decision trees, we can choose the features and the thresholds that directly optimize for classification accuracy! The point is we want to select the feature and threshold that maximally reduces label uncertainty, rather than selecting arbitrary feature and splitting at the median.
http://www.cs.columbia.edu/~verma/classes/ml/lec/lec1_intro_mle_bayes_naive_evaluation.pdf
The drawback of decision tree
We can overfit the data, which means the model gets used to the training data too much so that the error rate goes up.
http://www.cs.columbia.edu/~verma/classes/ml/lec/lec1_intro_mle_bayes_naive_evaluation.pdf
In order to alleviate the issue, we stop splitting the tree in the testing phase if the error rate goes up.