Data Classification Algorithms— Supervised Machine Learning at its best
By: Günter Röhrich
Tags: Computer Science, Machine Learning, Classification, Data Analysis, Data Science

Classification in general

Enough of the groundwork. The following parts of this article cover different approaches to separate data into, well, classes. Classification in an analytics sense is no different to what we understand when talking about classifying things in real life. In general, there are different ways of classification:

Binary classification : The possible response values can be e.g. “good” or “bad” — but in any way dichotomous.

: The possible response values can be e.g. “good” or “bad” — but in any way dichotomous. Multi-class classification: The possible outcomes can be multiple classes like grades, e.g. classifying a text as positive, negative, or neutral.

Multi-class classification is an exciting field to follow, often the underlying method is based on several binary classifications.

Support Vector Machines

SVMs rely on so-called support vectors, these vectors can be imagined as lines that separate a group of data points (a convex hull) from the rest of the space. Image two areas of data points that are clearly separable through a line, this is a so called “hard” classification task. The data points allow us to draw a straight line between the two “clusters” of data. The only problem we face is to find the line that creates the largest distance between the two clusters — and this is exactly what SVM is aiming at.

Hard Classification Task for SVM [1]

Hard Classification Task

Illustration 1 shows two support vectors (solid blue lines) that separate the two data point clouds (orange and grey). Our separator is the dotted line in the middle (which is interesting, as this actually isn’t a support vector at all). Finding the best separator is an optimization problem, the SVM model seeks the line that maximize the gap between the two dotted lines (indicated by the arrows), and this then is our classifier.

A side note, as the hard classification SVM model relies heavily on the margin-creation-process, it is of course quite sensitive to data points closer to the line rather than the points we see in the illustration. This is quite the inverse behavior compared to a standard regression line, where a closer point is actually less influential than a data point further away.

Soft Classification Task

The other way to use SVM is applying it on data that is not clearly separable, is called a “Soft” classification task. The soft SVM is based on not only the margin assumption from above, but also the amount of error it tries to minimize. In other words, soft SVM is a combination of error minimization and margin maximization.

Hard Classification would never work here [2]

Illustration 2 shows the case for which a hard classifier is not working — I have just re-arranged a few data points, the initial classifier is not correct anymore. The data points are not clearly separable any longer, hence we need to come up with a model that allows errors, but tries to keep them at a minimum — the soft classifier.

For both SVM approaches there are some important facts you must bear in mind:

Kernels do not have to be linear! There are many, many non-linear kernels you can use in order to fit data that cannot be properly separated through a straight line. SVM models provide coefficients (like regression) and therefore allow the importance of factors to be analyzed. SVM can be used for multi-class classification. As mentioned earlier, this approach can be boiled down to several binary classifications that are then merged together. Hard SVM classification can also be extended to add or reduce the intercept value. Depending on the price of a wrong classification, we might set the classifier at a slightly adjusted value (which is parallel to the one we originally calculated). Soft SVM allows to additionally add an error weight to the data points, e.g. data points for records 1–10 are less important than records 80–90. For this reason, the error for the points may be weighted differently.

K-Nearest Neighbors

Another non-parametric approach to classify your data points is k nearest neighbors (or short KNN). This is a pretty straight forward method to classify data, it is a very “tangible” idea of classification when it comes to several classes.

Scatter Plot with 4 different label/response values [3]

This method is not solving a hard optimization task (like it is done eventually in SVM), but it is often a very reliable method to classify data.

KNN is lazy. You will also not obtain coefficients like you would get from a SVM model, hence there is basically no real training for your model. KNN needs to look at the new data point and place it in context to the “old” data — this is why it is commonly known as a lazy algorithm.

New Pink Dot — likely to be Black [4]

As the illustration above shows, a new pink data point is added to the scatter plot. Measuring the distance from this new point to the closest 3 points around it, will indicate what class the point should be in. Usually, you would consider the mode of the values that surround the new one.

KNN is most commonly using the Euclidean distance to find the closest neighbors of every point, however, pretty much every p value (power) could be used for calculation (depending on your use case). You may have heard of Manhattan distance, where p=1 , whereas Euclidean distance is defined as p=2.

Looks fancy, but is actually just Pythagorean theorem (p=2)

However, there is one remaining question, how many values (neighbors) should be considered to identify the right class? It is recommended to test a few and see how they perform in terms of their overall model accuracy.

If you wanted to have a look at the KNN code in Python, R or Julia just follow the below link. The included GitHub Gists can be directly executed in the IDE of your choice:

Also note, that it might be wise to do proper validation on your results otherwise you might end up with a really bad model for new data points (variance!).

What you should not miss:

There is also the idea of KNN regression. In this case you will not see classes/labels but continuous values. Instead of assigning the label of the k closest neighbors, you could take an average (mean, µ), weighted averages, etc. of the related data points. KNN is not K-means, there are no cluster centers, but we use distances among related points. If this sounds confusing to you, never mind, jump back to the topic of supervised versus unsupervised learning. If you need a model that tells you what input values are more relevant than others, KNN might not be the way to go. KNN however is a straightforward and quite quick approach to find answers to what class a data point should be in.

Tree-based approaches

Tree-based models (Classification and Regression Tree models— CART) often work exceptionally well on pursuing regression or classification tasks. The main idea behind the tree-based approaches is that data is split into smaller junks according to one or several criteria. In tree jargon, there are branches that are connected to the leaves. The overall goal is to create branches and leaves as long as we observe a “sufficient drop in variance” in our data. If this is not the case, we stop branching.

If this sounds too abstract, think of a dataset containing people and their spending behavior, e.g. classify whether the person is in the target group or not (binary classification). Our intuition would probably look at the income first and separate data into a high- and low-income groups, pretty much like this:

How our tree might split the data into (arbitrarily chosen) portions

There might be many splits like this, maybe looking at the age of the person, maybe looking at the number of children or the number of hobbies a person has, etc. All these criteria may cause the leaf to create new branches having new leaves dividing the data into smaller junks.

If this striving for smaller and smaller junks sounds dangerous to you, your right — having tiny junks will lead to the problem of overfitting. For this reason, every leaf should at least have a certain number of data points in it, as a rule of thumb choose 5–10%.

In general, it is wise not to use all the available data to create the tree, but only a partial portion of the data— sounds familiar, right? This allows us to use the second dataset and see whether the data split we made when building the tree has really helped us to reduce the variance in our data — this is called “pruning” the tree.

Using R to build and prune a tree model: Lowest number of misclassifications occurs at k=7

In theory, we are using the second data portion to verify, whether the splits hold for other data as well, otherwise we remove the branch as it does not seem to provide sufficient benefit to our model.

In practice, the available libraries can build, prune and cross validate the tree model for you — please make sure you correctly follow the documentation and consider sound model selections standards (cross validation). The huge advantage of the tree model is, that for every leaf, we get the classifier’s (or regression’s) coefficients. These not only allow us to predict the outcome, but also provide insight into their overall importance to our model.

One tree, several trees, one Random Forest

Random forests (RF) can be summarized as a model consisting of many, many underlying tree models. Random forests consider a variety of different and randomly created, underlying trees and choose the most common response value. In other words, the random forest takes the mode out of all the responses predicted by the underlying tree models (or mean response in case of a regression random forest).

You can always plot the tree outcome and compare results to other models, using variations in the model parameters to find a fast, but accurate model:

My RF model in R — 200 trees should be enough

Stay with me, this is essential to understand when ‘talking random forest’: Using the RF model leads to the draw back, that there is no good way to identify the coefficients’ specific impact to our model (coefficient), we can only calculate the relative importance of each factor — this can be achieved through looking at the the effect of branching the factor and its total benefit to the underlying trees.

Regression for Classification?

I always wondered whether I could simply use regression to get a value between 0 and 1 and simply round (using a specified threshold) to obtain a class value. Well, this idea seemed reasonable at first, but as I could learn, a simple linear regression will not work. The reason for this is, that the values we get do not necessarily lie between 0 and 1, so how should we deal with a -42 as our response value?

The red zone: Impossible values for our classification

As you can see in the above illustration, an arbitrary selected value x={-1, 2} will be placed on the line somewhere in the red zone and therefore, not allow us to derive a response value that is either (at least) between or at best exactly 0 or 1. This is where the Sigmoid function comes in very handy. This function is commonly known as binary or logistic regression and provides probabilities ranging from 0 to 1.

The huge advantage is that even an infinitely small number is mapped to “close to” zero and will not be somewhere beyond our boundary. In the illustration below, you can find a sigmoid function that only shows a mapping for values -8 ≤ x ≤ 8.

The real deal when using regression for classification tasks — guaranteed red-zone-free

Having shown the huge advantage of logistic regression, there is one thing you need to keep in mind: As this model is not giving you a binary response, you are required to add another step to the entire modeling process. This means, it is necessary to specify a threshold (“cut-off” value) to round probabilities to 0 or 1 — think of 0.519, is this really a value you would like to see assigned to 1?

This clearly requires a so called confusion matrix. This matrix is used to identify how well a model works, hence showing you true/false positives and negatives. You could even get creative and assign different costs (weights) to the error type — this might get you a far more realistic result. E.g. the classification error of “the model says healthy, but in reality sick” is very high for a deadly disease — in this case the cost of a false positive may be much higher than a false negative.

A simple example of a confusion matrix

If you’re an R guy, caret library is the way to go as it offers many neat features to work with the confusion matrix.

There is one HUGE caveat to be aware of: Always specify the positive value (positive = 1), otherwise you may see confusing results — that could be another contributor to the name of the matrix ;)

A confusion matrix illustration

An important side note: The sigmoid function is an extremely powerful tool to use in analytics — as we just saw in the classification idea. Recently, there has been a lot of buzz going on around neural networks and deep learning, guess what, sigmoid is essential.

If you think of weights assigned to neurons in a neural network, the values may be far off from 0 and 1, however, eventually this is what we eventually wanted to see, “is a neuron active or not” — a nice classification task, isn’t it? Exactly here, the sigmoid function is (or actually used to be; pointer towards rectified linear unit) a brilliant method to scale all the neurons’ values onto a range of 0 and 1.

What you need to know about the logistic regression:

It provides you with probabilities ranging between 0 and 1, no class labels. Probabilities need to be “cut-off”, hence, require another step to conduct. Make sure you play around with the cut-off rates and assign the right costs to your classification errors, otherwise you might end up with a very wrong model. Working directly with the model coefficients is tricky enough (these are shown as log(odds) !). You are required to translate the log(odds) into probabilities. This might look familiar: p = log(odds) / [1 + log(odds)] In order to identify the most suitable cut-off value, the ROC curve is probably the quickest way to do so. ‘The ROC curve (receiver operating characteristic curve) is a graph showing the performance of a classification model at all classification thresholds.’ [source] AUC (area under the curve) provides the aggregated measure of performance of the model, given the underlying threshold. An AUC of 0.5 is pretty much guessing, whereas 1 means that predictions are 100% correct. Note, ROC is no substitute for the weight-assigned confusion matrix. Glad you’re asking: Yes, logistic regression can also be used in the course of CART model! Therefore, all the above mentioned concepts of trees apply to a Logistic CART too!

Deep Learning Network Classification

Deep learning networks (which can be both, supervised and unsupervised!) allow the classification of structured data in a variety of ways. Related methods are often suitable when dealing with many different class labels (multi-class), yet, they require a lot more coding work compared to a simpler support vector machine model. I will cover this exciting topic in a dedicated article. Meanwhile, a brilliant reference can be found here: