How to Train Your Model: A novice’s guide to selecting the correct machine learning algorithm for your problem
By: Adrian Yijie Xu
Tags: Beginners Guide, Machine Learning, Scikit Learn, Python, Beginner

Introduction

With the newfound proliferation and publicity of AI in science and industry, machine learning’s reputation as mystic art to data science problems has also grown exponentially. In reality, the proliferation of open-source packages such as Python’s Scikit-learn has led to a wave of democratization in the field, and made the implementation of machine learning techniques in your own datasets more facile than ever.

However, choosing the correct algorithm for your situation often devolves into trial and error as these packages come standard with multiple algorithms and little context. This approach, while comprehensive, is time-intensive and unscientific. The science lies in understanding how each individual algorithm works, and learning to pick the correct one for your situation. Understanding your tools raises your own capabilities as a data scientist, and prepares you for the development of your own solutions in the future.

In essence, a ML algorithm tries to infer relationships within your data. For instance, If you’ve ever plotted a line of best fit in math class, you already understand the principles of linear regression, an algorithm for continuous value prediction. You took your feature (on your x-axis) and your target (on your y-axis) and found a relationship between them that would allow you to extrapolate new values of y given some value of x.

However, there is no single algorithm that works for every situation. The algorithm choice depends on the amount and quality of data, the correlation of its features, as well as the task at hand. It must be emphasized that with enough high-quality data, any shortcomings in a single algorithm can be minimized.

In this article, we’ll walk you through how to choose an algorithms covering 4 different applications. To this effect, we assume that the user is familiar with the Python language and the Matplotlib package for visualization.

Theory

Let’s start by taking a look at Scikit-learn’s algorithm “cheat sheet”, which provides a great overview of the tools at hand.

A spider’s web through Scikit-Learn’s repetoire

Every data science problem is qualified by some key objectives. What kind of inference are you hoping to see in your data? Is the aim to classify a result into categories, to predict a continuous value, or to simply observe patterns within the data? Let’s briefly introduce each class.

Classification algorithms are used to predict whether a dataset’s outputs can be separated into separate classes, binary or otherwise. These output values are discrete and represent target classes. Classification algorithms undergo supervised training, which means they require labelled true output data in order to measure prediction accuracy.

are used to predict whether a dataset’s outputs can be separated into separate classes, binary or otherwise. These output values are discrete and represent target classes. Classification algorithms undergo supervised training, which means they require labelled true output data in order to measure prediction accuracy. Clustering algorithms can also be used for classification or simply to observe data patterns. By observing how the data is arranged within the feature space, clustering algorithms can utilize physical separation to create clusters. As such, some algorithms of this class don’t require output labels, making them unsupervised algorithms.

can also be used for classification or simply to observe data patterns. By observing how the data is arranged within the feature space, clustering algorithms can utilize physical separation to create clusters. As such, some algorithms of this class don’t require output labels, making them unsupervised algorithms. To predict a specific value in a continuous distribution given a set of inputs, regression algorithms relying on supervised training are often used. These also require labelled true output data in order to measure prediction accuracy.

relying on supervised training are often used. These also require labelled true output data in order to measure prediction accuracy. Dimensionality reduction algorithms focuses on decreasing the number of features from your dataset, preventing your models from “overfitting” or generalizing on previously unseen data. They are also unsupervised.

With our objectives laid out, let’s consider some specific user cases and solutions!

You’re trying to classify a discrete output variable given a limited amount of input data.

Logistic Regression

A logistic regression model computes a weighted sum of the input features together with a bias term, before utilizing a sigmoid function in order to scale the output into a number between 0 and 1. An illustration of a sigmoid fit to a dataset of two classes is shown below:

Logistic regression fit to a binary dataset. Source: Datacamp

This value represents the possibility that your output belongs to a particular target class. These coefficients are designed to best fit over the entire dataset by minimizing a defined sum or cost function:

Logistic Regression cost function. Source: Andrew Ng

You’ll observe that a misclassification for a single sample will result in an increase in the overall cost function. The aim of the network is to minimize this cost function, or in other words, to classify correctly according to true outputs.

Logistic regression models are simple to use, and work well with datasets featuring non-complex relationships. They scale well for small and medium-sized datasets.

Decision Trees (and random forests)

Decision tree-based models operate by doing a series of intra-model attribute classifications, where each row of data is evaluated with a series of conditions. When building a decision tree, we aim to keep the number of “splits” within the tree at a minimum, or in other words, to maximize the overall information gain. As a subset of decision trees, random forest classifiers build consensus through evaluating the outputs of an ensemble of decision trees, where the largest majority output class is taken as the final output.

Decsion tree based models tend to be better at learning more complex, non-linear relationships. Due to their reliance on individual attribute classification at each split, they do not mandate data preparation techniques such as feature scaling or centering nature. However, complex tree models are prone to overfitting.

Support Vector Machines

Support vector machine algorithms aim to locate a hyperplane in an N-dimensional space (where N refers to the number of features in your dataset) that distinctly classifies the data points, by maximizing the Euclidean separation of the two. Intuitively, you can approximate the SVM as attempting to building the widest possible path between two groups. The support vectors themselves are derived from the data points closest to the hyperplane, that assist in determining its orientation and position in order to maximize the margin of the classifier, ensuring good classification accuracy.

An example of SVM classifcation, with H3 being the ideal hyperplane for this dataset. Source: WIkipedia

Though an older concept, they’re still highly effective at classifying small and medium datasets, but do not provide any class probabilities for evaluation.

You’re trying to predict a continuous output variable given a limited amount of input data.

Linear Regression

Linear Regression falls into two sub-categories: single & multivariate. Single-variable linear regression can intuitively be thought of as plotting a line of best fit (in other words, one that minimizes the mean square error) between a single input feature variable and an output variable. For more complex datasets, multivariate linear regression attempts to do the same, but across the all of the dimensions of the feature data. The resulting linear relationship can be then used to predict an output value by computing a weighted sum of the input features together with a bias term, in a similar manner to logistic regression.

Linear regression is intuitive, robust, and works well non-complex datasets. They scale well for medium-sized datasets and above.

You’re trying to classify or predict a continuous output variable given large amounts of input data.

Neural Networks

A Neural network consists of layers containing groups of computational nodes called neurons, where every neuron in a layer is connected to every neuron in next layer. When a neuron receives some input value, it multiplies it with a weight and adds a bias value, followed by the application of a non-linear function. This non-linearity function gives the neural network the ability to model complex non-linear relationships observed within your data. After moving through the layers of the networks,, the individual neuron outputs are summed at a final output node, and a sigmoid or a linear function are added for classification or regression tasks, respectively. Neural networks are trained using optimizers or through an iterative process of gradient descent, where the weights of networks are tweaked to minimize the difference (often measured by mean squared error) between predicted and true outputs.

An example of a fully connected neural network with two output neurons. Source: Wikipedia

Intuitively, you can think of a neural network as an advanced implementation of a logistic regression model. While some of the cutting-edge work done in deep learning are done with them, complex neural networks can be prone to overfitting without proper compensation, and require large amounts of data to maximize performance (which in turn, increases computation time). Due to their complexity, feature importance evaluation can prove to be difficult.

You’ve a large amount of data, and want to observe if they form groups within the input feature space.

K-means

K-means algorithms attempt to group the datapoints into K-predefined classes via an iterative assignment process. Briefly, at each iteration, K points are assigned as cluster centers, representing the target classes. Each datapoint is then assigned to its closest cluster center, based on Euclidean distance. At the end of the iteration, the mean of all the distances within each cluster is calculated, and then minimized across several future iterations.

An example of K- means clustering at iteration 11. The centroids chosen here for the red, blue, and yellow datapoints are represented with a +, o , and x symbols, respectively. Source: Wikipedia

While by definition, an optimum will be reached using the K-means algorithm, it may not be the global optimum, requiring you to attempt multiple runs using new randomized centroids.

You think the high number of features in your dataset are causing overfitting in your models, and want to reduce the dimensionality of your data.

Principal Correlation Analysis

PCA is a statistical technique aiming to reduce the dimensionality of your data. While the comprehensive mathematical explanation of PCA is beyond the scope of this tutorial (add link here), briefly, PCA works by identifying principal component vectors, or those that do not exhibit correlation with your data and possess high variance. The component vector with maximum variance is then chosen as a plane, and all data projected onto it. This can illustrated as follows:

Identifying a principle component vector through a dataset, with criteria.

Intuitively, you can think of a PCA example as taking a 2D plot, identifying a plane running through its data with maximum variance, and projecting the datapoints onto this plane, resulting in a 1D plot. In the process, you’ve reduced the number of features by 1.

Implementation

As the documentation for Scikit-learn is extensive and comprehensive, we will only implement a simple binary classification example using logistic regression for reference. We’ll use Scikit-learn’s internal IRIS dataset, which contains the sepal and petal length data for three species of Iris- namely Setosa, Versicolour, and Virginica. These three classes are represented with separate label values of 0,1, and 2 within the dataset.

Let’s start by importing all of the libraries we need

from sklearn.linear_model import LogisticRegression

from sklearn import datasets

import numpy as np import matplotlib

import matplotlib.pyplot as plt

Next, let’s load our dataset. The iris dataset is particularly good as an introduction into ML, as it comes preloaded into the Scikit-Learn library. We split the dataset into a feature array (X), and a target label array (Y). In this example, we will use the petal length and width features.

iris = datasets.load_iris() X = iris[“data”][:,(2,3)] # petal length, petal width

y = iris[“target”]

Next, let’s define our classifier. We specify our three-class case and the optimization algorithm (LGBFS). The C parameter here relates to regularization — intuitively, it prevents the model from overfitting, which would lower it’s ability to generalize. Training this model (.fit) should take less than a second, after which let’s invoke the .score method to get a mean accuracy value over our data.

classifier = LogisticRegression(multi_class=”multinomial”, solver=”lbfgs”, C=5)

classifier.fit(X,y)

print(classifier.score(X,y))

Finally, let’s plot the results.

X_new = np.linspace(0,5,100).reshape(-1,10) plt.plot(X[:, 0][y==1], X[:, 1][y==1], “y.”, label=”Iris-Versicolor”)

plt.plot(X[:, 0][y==0], X[:, 1][y==0], “b.”, label=”Iris-Setosa”)

plt.plot(X[:, 0][y==2], X[:, 1][y==2], “r.”, label=”Iris-Virginica”) plt.legend(loc=”upper left”, fontsize=14) plt.show()

Running the code above yields a respectable mean classification accuracy of 96% using only a training dataset, and our plot clearly shows a successful separation of our data into the three classes. Go ahead and see if you can improve upon this value using a Random Forest Classifier!

Output plot of our logistic regression classifier trained on the Iris dataset.

Hopefully, you’ve found this tutorial useful, and are inspired to explore the use of machine learning in your own data. For any specific questions, feel free to leave a comment!

References

Various, Harvard CS109 — Data Science

Aurélien Geron, Hands-On Machine Learning with Scikit-Learn and TensorFlow

Andrew Ng, Machine Learning