Unsupervised Learning: Contemporary Classification of Machine Learning Techniques (Part 2)
By: Donny Soh
Tags: Unsupervised Learning, Machine Learning, Artificial Intelligence, Technology

In this installment of the series titled Learning Machine Learning, we will move on to the next class of machine learning techniques: non-supervised learning. If you would like to read my previous take on supervised learning, you can find it here.

Characters exploring their universe without supervision

Contemporary Classification of Machine Learning Algorithms

As mentioned, today’s Machine Learning Algorithms can be segregated into one of the three classes, supervised learning, unsupervised learning and reinforcement learning. This segregation is chosen because of the way these algorithms learn the machine learning model. In particular:

Supervised learning: learns the model by example

Unsupervised learning: learns the model from data

Reinforcement learning: learns the model by experience

Unsupervised Learning: Learning from Data

In contrast to supervised learning where your training data is always labeled, data used in unsupervised learning methods have no classification labels.

So what happens during unsupervised learning is: a huge mass of training data gets passed into the learning algorithm. The unsupervised algorithm automatically extracts features from this huge mass of data which best represents/summarizes it.

So what happens during unsupervised learning is: a huge mass of training data gets passed into the learning algorithm. The unsupervised algorithm automatically extracts features from this huge mass of data which best represents/summarizes it.

Such unsupervised learners can be subdivided into various groups. On this, I would refer to Ruslan Salakhutdinov’s (CMU) division of unsupervised learners into two groups, the non-probabilistic and the probabilistic:

Figure 1: Unsupervised learners. Source: https://www.youtube.com/watch?v=rK6bchqeaN8

This article’s discussion will be constrained only to classical unsupervised approaches such as clustering and autoencoders. Probabilistic learners are beyond the scope of this article for now and hopefully, we will return to them another time.

Clustering

Clustering is a technique which automatically finds similar groups of objects in a dataset. However, the way this “similarity” is defined differs from situation to situation. In some situations, the proximity of data points may determine the best clusters, while in other situations, it might be the density of the data points. The image from sklearn illustrates this point really well.

Figure 2: Comparison of different clustering algorithms Source: sklearn

Using the first row from Figure 2 as an example, assume you have the data that looks like the following:

Figure 3: Original data that is unclustered

Looking at Figure 3, you would probably want the algorithm to give you two clusters, preferably one cluster from the inner circle and the other from the outer circle. If we refer back to the first row of Figure 2, you would find that most of the algorithms stated there would not provide you with such a clustering. In fact, only three of them would and they are spectral clustering, agglomerative clustering, and DBScan.

Figure 4: Correct clustering, identifying both an inner circle and an outer circle.

Notice that most of the other algorithms give you a split of the two circles in the middle. That is because these algorithms calculate clusters primarily based on the distance between points. So these algorithms would have calculated the center of these clusters (centroid) equally splitting data into the right and the left half-circles.

Figure 5: Incorrect clustering with centroids equally splitting the data on the left and right.

Now let us build upon the example from our first article. In that article, we used an example on building a decision tree on how one should go to work, based on certain conditions, such as if it is raining, or if the train has a breakdown. The data from the example is reproduced here in Table 1 for ease of reference.

Table 1: Training observations

If we were to plot the ground truth of the observation table (using just three features Rain, Train Breakdown and Oversleep), we will end up with this 3D map:

Figure 6: Ground truth of Table 1 projected on a 3D map.

Now we will compare these results with that of unsupervised learning. As mentioned for unsupervised learning, our observations have no labels. We first reproduce the same observation table but after removing the labels from the table.

Table 2: Training observations without classification labels

If we were to run a clustering algorithm such as k-means in Table 2, we end up with clusters that look like this in the same 3D map:

Figure 7: k-means clustering

Incidentally, when we compare the clusters k-means gives to the data, we can get a “classification accuracy” of 0.75. Loosely speaking, it means that we are able to use k-means to determine the clusters that properly determine if the data refers to that of a bus or a train.

If we were to use an alternative clustering algorithm which focuses on density and not distance, we will be able to get a slightly better “clustering accuracy” in our example. In fact, using Agglomerative Clustering gives an accuracy of 0.875. The cluster map is shown below:

Figure 8: Agglomerative clustering

One last final point in this example. If we were to use this concept of clustering and apply it to the decision tree we learned in Part 1, we would end up with a 2D map that looks like the following:

Figure 9: 2D cluster map of a decision tree

Autoencoders

The other class of unsupervised learning algorithms we discuss here are autoencoders. Autoencoders are used for feature extraction and these features are learned by training a neural network where the inputs and outputs are exactly the same. The typical structure of an autoencoder is shown in Figure 10.

Figure 10: Typical structure of an Autoencoder. Source

Autoencoders typically consists of three portions, the encoder (where data gets encoded to the bottleneck features), the decoder, where the bottleneck code features gets reconstructed back to the data and the bottleneck code features. Hence when we feed the input layer data A, we are expecting the output layer to return us reconstructed data A` where A` is as close to A as possible. (You can think of the optimization loss function to be |A` — A|).

A trivial solution is to have the neural network directly copy the inputs to the outputs, hence always returning a reconstructed A` which is exactly the same as A. However you would note that the bottleneck layer contains significantly fewer nodes than that of the input or output layers. This prevents the trivial situation from happening and because the bottleneck contains fewer nodes, the network forces the bottleneck to learn and summarize the key features of A to reconstruct the output layer A`.

Examples of areas where autoencoders are used are dimensional reduction, anomaly detection, and noise removal. Contemporary techniques for translation use a variant of autoencoders, known as seq-to-seq autoencoders. Another autoencoder variant known as word embeddings has been readily used in NLP applications.

I am currently preparing python code snippets to illustrate these examples and my subsequent articles on autoencoders will contain working demos to demonstrate these examples.

In the final article of this three-part series, we will conclude our discussion on classifications in machine learning algorithms with a primer on reinforcement learning.

The accompanying short video lecture series can be found on youtube here.

The author is an assistant professor at the Singapore Institute of Technology (SIT). He holds a PhD in Computer Science from Imperial College. He also has a Masters in Computer Science from the NUS under the Singapore MIT Alliance (SMA) programme.

The views in this article are that of the author’s and do not necessarily reflect the official policies or positions of any organizations that the author is associated with. The author also holds no affiliations nor earns any fees from any products, courses or books mentioned in this article.