Why We Use Unsupervised Learning (With K-means Clustering From Scratch) | by Emmett Boudreau | Towards Data Science

Unsupervised learning is an interesting topic in the Data-Science world that isn’t often highlighted by those who aren’t doing Data-Science, and furthermore is often an idea neglected by many Data-Scientists themselves. There is an explanation for this, as for many employment opportunities, unsupervised learning simply isn’t significant. There might certainly be times where it will come into play, but for the most part, a business is not going to invest in doing research where they will never know the results. Doing research whilst not knowing the results is exactly what the unsupervised adjective applies in this situation, so if we don’t know what our model is putting out, then how can we find value in its predictions and findings?
The primary function of unsupervised learning algorithms is analysis. Using an unsupervised learning algorithm to explore your data can tell you a lot about certain attributes of said data. Clustering, for example, can show how grouped certain continuous values might be, whether related or unrelated. You can use unsupervised learning to find natural patterns in data that aren’t immediately obvious with just statistical analysis or comparing values.
Unsupervised learning algorithms also hold their own in image recognition and genomics as well. In genomics, they can be used to cluster together genetics or analyse sequences of genome data. Unsupervised learning is used to model probability densities, which is incredibly useful to the Bioinformatics discipline.
Another great thing about unsupervised is that it can be incredibly easy to gather analytics without any human interaction whatsoever. Typical machine-learning, whether reinforced or supervised, will require manually drawn labels in order to properly understand the outcome of the model. The differences that distinguish supervised and unsupervised learning are as follows:
Less Accurate
No labeled data required
Minimal human effort
Highly accurate
Has a consistent target
Labeled data is required
Requires human effort in the loop
Clustering is the most commonly used unsupervised learning method. This is because typically it is one of the best ways to explore and find out more about data visually. There are several different types of clustering, including:
Hierarchical clustering: builds a multilevel hierarchy of clusters by creating a cluster tree.
k-Means clustering: partitions data into k distinct clusters based on distance to the centroid of a cluster.
Gaussian mixture models: models clusters as a mixture of multivariate normal density components.
Self-organizing maps: uses neural networks that learn the topology and distribution of the data.
Hidden Markov models: uses observed data to recover the sequence of states.
For today’s example, we are going to be looking at Kmeans clustering, which is available in Sklearn. In K-means clustering, for each point x:
Find the nearest centroid c
assign the point x to cluster j
and then for each cluster j (=1 .. K):
The new centroid c is equal to the mean of all points x assigned to the cluster j in the previous step.
While this model is available in sklearn.cluster under the class KMeans, today I am going to be writing my own function to calculate the K-means clusters. We’re going to start this by creating a function that has the ability to measure euclidian length. This is as simple as using linalg from Numpy.
import numpy as npdef euclidian(a, b):    return np.linalg.norm(a-b)
I also went ahead and imported Matplotlib.pyplot and Matplotlib.animation for some visualizations of our clusters in the future:
import matplotlib.pyplot as plt# animationimport matplotlib.animation as animation
Since I’m going to be using text data, I used this function from numpy to load it:
def load_dataset(name):    return np.loadtxt(name)
Now it’s time to make our actual K-means function. To start, we will add the parameters, k, epsilon, and distance (which will be euclidean in our case.) After that, we will make an empty list for our centroids, and setup the distance algorithm to use our euclidian function from before.
def kmeans(k, epsilon=0, distance='euclidian'):    history_centroids = []    #set the distance calculation type     if distance == 'euclidian':        dist_method = euclidian
Next, we will load our data-set and check the data-set’s shape to get the number of instances (observations), and the number of features
dataset = load_dataset('durudataset.txt')num_instances, num_features = dataset.shape
Now we will define our k-prototypes using a random number from Numpy.random:
prototypes = dataset[np.random.randint(0, num_instances - 1, size=k)]
After that, we will append them to our centroid history list:
history_centroids.append(prototypes)
Now I created these lists to store our centroid clusters and keep track of them at each iteration:
prototypes_old = np.zeros(prototypes.shape)belongs_to = np.zeros((num_instances, 1))norm = dist_method(prototypes, prototypes_old)iteration = 0
Then we will do our for loop, following the formula I stated before:
while norm > epsilon:        iteration += 1        norm = dist_method(prototypes, prototypes_old)        for index_instance, instance in enumerate(dataset):            dist_vec = np.zeros((k,1))            for index_prototype, prototype in enumerate(prototypes):                #compute the distance between x and centroid                dist_vec[index_prototype] = dist_method(prototype, instance)            belongs_to[index_instance, 0] = np.argmin(dist_vec)                    tmp_prototypes = np.zeros((k, num_features))
While the norm is greater than epsilon, for each instance in the data-set we will define a distance vector that is the size of k. Then for each centroid, we will calculate the difference between our x and the centroid. Next, we will loop through our list of prototypes and get all of the points that are assigned to said prototype. Then we will find the mean of those points, which will give us our new centroid. Last but not least, we will add that corresponding value to the index of our list.
for index in range(len(prototypes)):            instances_close = [i for i in range(len(belongs_to)) if belongs_to[i] == index]            prototype = np.mean(dataset[instances_close], axis=0)            #add our new centroid to our new temporary list            tmp_prototypes[index, :] = prototype
Finally, we can set our temporary prototypes equal to our final list of centroids. For the sake of creating a Matplotlib animation, I will also add the history of centroids list to the mix here.
prototypes = tmp_prototypeshistory_centroids.append(tmp_prototypes)return prototypes, history_centroids, belongs_to
For a final function that looks like this:
def kmeans(k, epsilon=0, distance='euclidian'):    history_centroids = []    if distance == 'euclidian':        dist_method = euclidian    #set the dataset    dataset = load_dataset('durudataset.txt')    num_instances, num_features = dataset.shape     prototypes = dataset[np.random.randint(0, num_instances - 1, size=k)]    history_centroids.append(prototypes)    prototypes_old = np.zeros(prototypes.shape)    belongs_to = np.zeros((num_instances, 1))    norm = dist_method(prototypes, prototypes_old)    iteration = 0    while norm > epsilon:        iteration += 1        norm = dist_method(prototypes, prototypes_old)        for index_instance, instance in enumerate(dataset):            dist_vec = np.zeros((k,1))            for index_prototype, prototype in enumerate(prototypes):                dist_vec[index_prototype] = dist_method(prototype, instance)            belongs_to[index_instance, 0] = np.argmin(dist_vec)                    tmp_prototypes = np.zeros((k, num_features))        for index in range(len(prototypes)):            instances_close = [i for i in range(len(belongs_to)) if belongs_to[i] == index]            prototype = np.mean(dataset[instances_close], axis=0)            #add our new centroid to our new temporary list            tmp_prototypes[index, :] = prototype        prototypes = tmp_prototypes        history_centroids.append(tmp_prototypes)    return prototypes, history_centroids, belongs_to
Now if we decide to plot it, my results looked a little something like this:
Pretty cool, right?
While unsupervised learning might not get the love or the use that most supervised learning models enjoy, just because the results aren’t labeled doesn’t mean that there can’t be a lot of information learned about the data from it. Unsupervised learning is a great tool for exploring and truly understanding how data is grouped and how different features interact with one another. Although in a lot of cases, Data-Scientists might stray a bit far away from using unsupervised learning, it’s easy to see why it could be quite beneficial sometimes!