Time and Space Complexity of Machine Learning Models
By: Chandrasekhar Gundluru
Tags: AI, Machine Learning, Logistic Regression, Space Complexity, Time Complexity

The train time complexity of machine learning model — The amount of time taken to train the model

— The amount of time taken to train the model The test time complexity of the machine learning model — Time took to predict output for a given input query point.

Time complexity is an essential aspect to know when anyone wants their model with low latency. Let’s dive deep into details of how much time and space required by the wide variety of models to predict the output.

“Assuming training data has n points with d dimensions “

1) K-Nearest Neighbors:

Given query point (xq), K-NN follows these steps to predict output (yq). For each xi in the training set, Compute distance for xi and xq.

Taken from https://www.edureka.co/blog/k-nearest-neighbors-algorithm/

Computing distance for one point takes o(d) time, as we are computing for each xi in training it takes o(nd)

Then store the smallest k distances in a list and the Majority vote to K points, which took o(1). As we need whole train data during run time, Space complexity will be o(nd) finally for KNN

Run Time complexity = O(nd) Space complexity = O(nd)

There is no train time complexity for KNN

2) Naive Bayes:

Given query point (xq), Naive Bayes need just prior and likelihood probabilities to predict output (yq), which permits the model to use at low latency applications.

the pic was taken from https://heartbeat.fritz.ai/understanding-the-mathematics-behind-naive-bayes-ab6ee85f50d0

So, during the training phase of Naive Bayes, it calculates all likelihood probabilities and prior probability, which takes Time complexity of O(ndc) where c= Number of classes and Space complexity= O(dc)

Train Time complexity = O(n*d*c) Run Time complexity =O(d*c) Space complexity = O(d*c)

3 ) Logistic Regression:

For query point (xq), Logistic Regression needs just a d-dimension weight vector(W).

yq = sigmoid(W.xq), which takes O(d) time and space required is O(d)

For the Training Logistic Regression, the optimization problem needs to be solved using Stochastic Gradient Descent that takes O(nd).

Logistic Regression is applicable for low latency applications and memory-efficient at run time.

Train Time complexity = O(n*d) Run Time complexity =O(d) Space complexity = O(d)

4) Decision Tree:

Train Time complexity = O(n*log(n)*d) Space complexity=O(p) where p= no of nodes in tree Run Time complexity= O(k) where k= depth of tree

why O(n*log(n)*d)?

Let’s say the dataset has n points d features and all features are numerical.

As the numerical feature needs to be sorted, Sorting one feature takes O(n*lgn)

Then Sorting all d features will take O(n*lgn*d)

Now we need to calculate Information gain at each threshold. For one feature, it takes O(n), and for d features, it takes O(n*d)

Time complexity is O(n*lgn*d)+ O(nd)

Because of Big O notation, O(nlogn*d)+O(n*d) will be almost equal to O(nlogn*d).

Runtime space complexity =o(no. of nodes).

During Runtime for DT, it only needs if-else conditions for decision

thus space complexity is reasonable.

5) Bootstrap Aggregation with Decision Trees (or) Random forest :

As the random forest is formed from k base learners (Decision Trees) for query point xq to predict yq we get k outputs then by Aggregation (majority weight or mean/median) is applied on the outputs of the k base learners (DTs) to generate yq.

for the random forest with m decision trees

Train Time complexity = O(n*log(n)*d*m) Space complexity=O(p*m) Run Time complexity= O(k*m)

During Training, Random Forest can be parallelized as each base learner can be trained on the different core of the computer.

6) Gradient Boost Decision Tree :

Train Time complexity = O(n*log(n)*d*m) Space complexity=O(p*m+gamma) as has to be multiplied with m models. Run Time complexity= O(k*m)

7) SVM:

Train Time complexity = O(n²) Space complexity=O(s) Run-time Complexity= O(s*d)

s= number of Support Vectors,d=dimentionality of the data

8) References:

www.appliedaicourse.com

https://www.hackerearth.com/practice/basic-programming/complexity-analysis/time-and-space-complexity/tutorial/#:~:text=Time%20complexity%20of%20an%20algorithm,the%20length%20of%20the%20input.&text=Let%20each%20operation%20takes%20time.