Log Book — Guide to Distance Measuring Approaches for K- Means Clustering | by dearC | Towards Data Science

Let’s begin with a brief introduction of clustering. Clustering is the task of dividing the data points into a number of groups such that data points in the same groups are more similar to other data points in the same group than those in other groups. In simple words, the aim is to segregate groups with similar features and assign them into clusters.
K-Means Clustering is one of the many clustering algorithms. The idea behind it is to define clusters so that the total intra-cluster variation (known as total within-cluster variation) is minimized. K-means algorithm can be summarized as follow:
1. Specify the number of clusters(k) to be created.2. Select randomly k objects from the data-set as the initial cluster centers or means.3. Assign each observation to their closest centroid, based on the specified distance[the type of distance is what we will be exploring in this article, in the above case it is Euclidean] between the object and the centroid.4. For each of the k clusters update the cluster centroid by calculating the new mean values of all the data points in the cluster. The centroid of a K-th cluster is a vector of length p containing the means of all variables for the observations in the K-th cluster; p is the number of variables.5. Iteratively minimize the total within sum of square. That is, iterate steps 3 and 4 until the cluster assignments stop changing or the maximum number of iterations is reached.
In R we calculate the K-Means cluster by:
Kmeans(x, centers, iter.max = 10, nstart = 1, method = "euclidean")wherex        > Data framecenters  > Number of clustersiter.max > The maximum number of iterations allowednstart   > How many random sets of center should be chosenmethod   > The distance measure to be usedThere are other options too of calculating kmeans clustering but this is the usual pattern.
There are different distance calculation methods like euclidean, maximum( Chebychev Distance), manhattan, hamming, canberra, pearson , abspearson , abscorrelation, spearman or kendall. So how do you choose which one to use?
The methods are defined into 2 groups, one is based on capturing the geometrical separation and another one is dependent on correlation. We will take a look at each.
Euclidean, Manhattan & Maximum(Chebychev) distance
To begin with, a lot of this material in this section has been referred from the now offline page of divingintodatascience, the site had been of great help.
Minkowski distance is a metric which tells us the distance between 2 points in space. Now Minkowski distance come of different orders and we will shortly see what it means and we will also see why I am talking about it instead of Euclidean and other distances.
The generic formula for Minkowski distance for 2 points p and q:
is given by:
Minkowski distance is typically used with r being 1 or 2, which correspond to the Manhattan distance and the Euclidean distance respectively. In the limiting case of r reaching infinity, we obtain the Chebychev distance.
An easier way to understand is with the below picture
Manhattan distance captures the distance between two points by aggregating the pairwise absolute difference between each variable while Euclidean distance captures the same by aggregating the squared difference in each variable. Therefore, if two points are close on most variables, but more discrepant on one of them, Euclidean distance will exaggerate that discrepancy, whereas Manhattan distance will shrug it off, being more influenced by the closeness of the other variables. The Chebychev distance calculates the maximum of the absolute differences between the features of a pair of data points.
Manhattan distance should give more robust results, whereas Euclidean distance is likely to be influenced by outliers. Same applies to the higher values of “p” in Minkowski distance formula. As we increase the value of p, distance measure become more susceptible to lose the robustness and outliers in few dimensions start dominating the distance value.
An interesting observation can be made about the difference among these if we draw a ‘Circle’ using these different distance measures instead of default Euclidean one. As we know that a Circle is the locus of a point equidistant from a given point, the center of the circle. Now, if we use Manhattan or Chebychev distance measures to measure the distance of points from center, we actually get “squares” instead of usual “round-shape” circles.
Canberra Distance
It is a weighted version of Manhattan distance. It measures the sum of absolute fractional differences between the features of a pair of data points and is very sensitive to a small change when both coordinates are nearest to zero.
Hamming Distance
For categorical variables (male/female, or small/medium/large), we can define the distance as 0 if two points are in the same category, and 1 otherwise. If all the variables are categorical, then you can use Hamming distance, which counts the number of mismatches. You can also expand categorical variables to indicator variables, one for each level of the variable.If the categories are ordered (like small/medium/large) so that some categories are “closer” to each other than others, then you can convert them to a numerical sequence. For example, (small/medium/large) might map to (1/2/3). Then you can use Euclidean distance, or other distances for quantitative data.
Mahalanobis Distance
We can think of the Mahalanobis distance from a point to its respective cluster center as its Euclidean distance divided by the square root of the variance in the direction of the point. The Mahalanobis distance metric is preferable to the Euclidean distance metric because it allows for some flexibility in the structure of the clusters and takes into account variances and covariances amongst the variables.
When you use Euclidean distance, you assume that the clusters have identity covariances. In 2D, this means that your clusters have circular shapes. Obviously, if the covariances of the natural groupings in your data are not identity matrices, e.g. in 2D, clusters have elliptical shaped covariances, then using Mahalanobis over Euclidean will be much better modeling.
Correlation-based distance considers two objects to be similar if their features are highly correlated, even though the observed values may be far apart in terms of geometrical distance. The distance between two objects is 0 when they are perfectly correlated. If you want to identify clusters of observations with the same overall profiles regardless of their magnitudes, then you should go with correlation-based distance as a dissimilarity measure.
If Euclidean distance is chosen, then observations with high values of features will be clustered together. The same holds true for observations with low values of features.
Pearson correlation distance
Pearson correlation measures the degree of a linear relationship between two profiles. Pearson correlation analysis is the most commonly used method. It is also known as a parametric correlation which depends on the distribution of the data. This distance is based on the Pearson correlation coefficient that is calculated from the sample values and their standard deviations. The correlation coefficient ‘r’ takes values from –1 (large, negative correlation) to +1 (large, positive correlation).
There are a few other variants of this distance:
Absolute Pearson Correlation distance : In this distance, the absolute value of the Pearson correlation coefficient is used; hence the corresponding distance lies between 0 and 1.Un-centered Correlation distance: This is the same as the Pearson correlation, except that the sample means are set to zero in the expression for un-centered correlation. The un-centered correlation coefficient lies between –1 and +1; hence the distance lies between 0 and 2.Absolute, Un-centered Correlation distance: This is the same as the Absolute Pearson correlation, except that the sample means are set to zero in the expression for un-centered correlation. The un-centered correlation coefficient lies between 0 and +1; hence the distance lies between 0 and 1.
Absolute Pearson Correlation distance : In this distance, the absolute value of the Pearson correlation coefficient is used; hence the corresponding distance lies between 0 and 1.
Un-centered Correlation distance: This is the same as the Pearson correlation, except that the sample means are set to zero in the expression for un-centered correlation. The un-centered correlation coefficient lies between –1 and +1; hence the distance lies between 0 and 2.
Absolute, Un-centered Correlation distance: This is the same as the Absolute Pearson correlation, except that the sample means are set to zero in the expression for un-centered correlation. The un-centered correlation coefficient lies between 0 and +1; hence the distance lies between 0 and 1.
Eisen cosine correlation distance
It’s a special case of Pearson’s correlation with x ̄ and y ̄ both replaced by zero:
Spearman & Kendall correlation distance
The Spearman correlation between two variables is equal to the Pearson correlation between the rank values of those two variables; while Pearson’s correlation assesses linear relationships, Spearman’s correlation assesses monotonic relationships (whether linear or not). If there are no repeated data values, a perfect Spearman correlation of +1 or −1 occurs when each of the variables is a perfect monotone function of the other.
Intuitively, the Spearman correlation between two variables will be high when observations have a similar (or identical for a correlation of 1) rank (i.e. relative position label of the observations within the variable: 1st, 2nd, 3rd, etc.) between the two variables, and low when observations have a dissimilar (or fully opposed for a correlation of −1) rank between the two variables.
The Kendall tau rank distance is a metric that counts the number of pairwise disagreements between two ranking lists. The larger the distance, the more dissimilar the two lists are. Kendall tau distance is also called bubble-sort distance since it is equivalent to the number of swaps that the bubble sort algorithm would take to place one list in the same order as the other list.
Spearman’s coefficient is appropriate for both continuous and discrete ordinal variables. Both Spearman’s ρ and Kendall’s τ can be formulated as special cases of a more general correlation coefficient.
The value of distance measures is intimately related to the scale on which measurements are made. Therefore, variables are often scaled before measuring the inter-observation dissimilarities. This is particularly recommended when variables are measured in different scales (e.g: kilograms, kilometers, centimeters, ...); otherwise, the dissimilarity measures obtained will be severely affected.Standardization makes the four distance measure methods — Euclidean, Manhattan, Correlation and Eisen — more similar than they would be with non-transformed data. Note that, when the data are standardized, there is a functional relationship between the Pearson correlation coefficient & Euclidean distance.k-means works with continuous variables. It should not be done with data of mixed types. When your data consist of variables of mixed types, you can try using Gower’s distance. There is an overview of Gower’s distance here.
The value of distance measures is intimately related to the scale on which measurements are made. Therefore, variables are often scaled before measuring the inter-observation dissimilarities. This is particularly recommended when variables are measured in different scales (e.g: kilograms, kilometers, centimeters, ...); otherwise, the dissimilarity measures obtained will be severely affected.Standardization makes the four distance measure methods — Euclidean, Manhattan, Correlation and Eisen — more similar than they would be with non-transformed data. Note that, when the data are standardized, there is a functional relationship between the Pearson correlation coefficient & Euclidean distance.
k-means works with continuous variables. It should not be done with data of mixed types. When your data consist of variables of mixed types, you can try using Gower’s distance. There is an overview of Gower’s distance here.
There is only a best distance measure for a given data-set. The choice of the distance measure WILL influence your clustering, but it depends on the data-set and on the objective, which distance measure is most adequate for your particular application.