Importance of Understanding the Complexity of a Machine Learning Algorithm | by Baran Köseoğlu | Towards Data Science

Machine learning engineers often find themselves in situations to choose the right algorithm for the problem at hand. It is often the case that they first understand the structure of the problem they provide a solution to. They then explore the dataset at hand. After initial observations and key takeaways, they finally choose the right algorithm for the task.
Deciding on the best algorithm to apply to the dataset at hand may seem a trivial job. Engineers often create shortcuts in these situations. If the task has 0–1 label, just apply logistic regression, right? Wrong! We should be aware of these shortcuts and always remind ourselves that although certain algorithms work good with specific problems, there is no “recipe” when selecting the best algorithm for the problem. However, complexity and runtime analysis of an algorithm should always be discussed and taken into account.
Runtime analysis of an algorithm is not only crucial to comprehend the inner workings of the algorithm but it also yields a more successful implementation.
In the rest of the article, I am going to depict a situation, where omitting to think about runtime analysis of an algorithm, k means clustering in this case, can cause an engineer to lose lots of time and energy.
K-means clustering is one of the most popular and easy-to-implement unsupervised machine learning algorithms. It is also one of the easy-to-comprehend machine learning algorithms.
Typically, unsupervised machine learning algorithms make inferences from the input dataset using only the feature vectors. Therefore these algorithms are good for datasets having no label data. They can also be quite useful when one wants to extract value or insight from large quantities of structured and unstructured data. K-means clustering is one of these exploratory data analysis techniques, where the goal is to extract subgroups of data points such that the data points in the same cluster are very similar in terms of the features defined.
K-means clustering starts with the first group of randomly selected data points, which are used as the initial seed of the centroids. The algorithm then performs iterative calculations to assign the rest of the data points to the closest cluster. While performing these calculations according to the distance function defined, the positions of the centroids are updated. It halts the optimizing the center of the clusters when either:
The positions of the centroids are stabilized, i.e the change in their values does not exceed a predefined threshold.
The algorithm surpassed a maximum number of iterations.
Therefore, the complexity of the algorithm is
O(n * K * I * d)n : number of pointsK : number of clustersI : number of iterationsd : number of attributes
I am going to share code snippets for a k-means clustering task. My only purpose is to demonstrate an example, where failing to comprehend runtime complexity results in a poor evaluation of the algorithm. The steps I took are not optimized for the algorithm, i.e you can preprocess the data better and get finer clusters. Outline of the steps involved is as follows:
Import libraries, and read dataset. Here I import relevant libraries and read the dataset, which I already downloaded to a local folder.Preprocessing. In this step, I discard the columns of string type and only focus on numeric features. Since k-means clustering calculates the distance between data points, it works with numeric columns.Applying PCA to reduce dimension. It is usually good practice to reduce the dimension of the dataset before applying k-means clustering, as in high dimensional spaces, distance measures do not work very well.Calculating silhouette score. K-means clustering not applied directly. It involves the problem of finding an optimal number of clusters. Silhouette score is one of the techniques you can use to determine an optimal number of clusters. Failing to understand the complexity of the calculations involved in silhouette score analysis will yield poor implementation.Alternative solutions. Here I list some of the alternative solutions to find an optimal number of clusters. They are advantageous compared to the silhouette score in terms of runtime complexity.
Import libraries, and read dataset. Here I import relevant libraries and read the dataset, which I already downloaded to a local folder.
Preprocessing. In this step, I discard the columns of string type and only focus on numeric features. Since k-means clustering calculates the distance between data points, it works with numeric columns.
Applying PCA to reduce dimension. It is usually good practice to reduce the dimension of the dataset before applying k-means clustering, as in high dimensional spaces, distance measures do not work very well.
Calculating silhouette score. K-means clustering not applied directly. It involves the problem of finding an optimal number of clusters. Silhouette score is one of the techniques you can use to determine an optimal number of clusters. Failing to understand the complexity of the calculations involved in silhouette score analysis will yield poor implementation.
Alternative solutions. Here I list some of the alternative solutions to find an optimal number of clusters. They are advantageous compared to the silhouette score in terms of runtime complexity.
You can reproduce the problem to give it a try. The link to the dataset: https://www.kaggle.com/sobhanmoosavi/us-accidents.
We only cluster data points according to road-related features for illustration purposes.
It seems 3 is optimal.
There are many indices and methods for identifying the optimal number of clusters. But I will focus a few of them. Silhouette score is one of these metrics. It is calculated using the mean intra-cluster distance and the mean nearest-cluster distance for each instance. It computes the distance between each sample and the rest of the samples in the respective clusters. Therefore its runtime complexity is O(n2). Failing to perform runtime analysis, you may need to wait hours if not days for the analysis to complete for a large dataset. Since the current dataset has millions of rows, a workaround can be using simpler metrics such as inertia or applying random sampling to the dataset. I will show these two alternatives.
Elbow method
This method uses inertia or within-cluster sum-of-squares as it’s input. It depicts the decrease in the inertia value in response to an increasing number of clusters. “Elbow” (the point of inflection on the curve) is a good indication of the point, where the decrease in the inertia value doesn’t change significantly. The advantage of using this technique is that the within-cluster sum-of-squares is not as computationally expensive as silhouette score and is already included in the algorithm as a metric.
The wall-clock time of the code snippet above is:
27.9 s ± 247 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Random downsampling
Downsampling allows you to work with much smaller datasets. The advantage is that the time it takes for an algorithm to finish decreases drastically. This enables the analyst to work faster. A downside is that downsampling, if done randomly, may not represent the original dataset. Therefore, any analysis involving downsampled datasets may result in inaccurate results. However, you can always take precautions to make sure that the downsampled dataset represents the original dataset.
The wall-clock time of the code snippet above is:
3min 25s ± 640 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)
Final Remarks
In this post, I tried to emphasize the importance of understanding the complexity of a machine learning algorithm. Runtime analysis of an algorithm is not only crucial for algorithm selection in a specific task but is also important for successful implementation. It is also one of the key skills that most employers look for in the field of data science. Therefore, it is always a good practice to do runtime analysis and comprehend the complexity of an algorithm.
If you have any questions regarding the post or any questions about data science in general, you can find me on Linkedin.