Right Model for the Job
By: Sakchhi Srivastava
Tags: Machine Learning, Predictive Analytics, Model Selection, Data Science

Right Model for the Job

Choosing the right machine learning model for a business use case can be tricky. With so many models easily accessible as part of freely available libraries and more being added every day, it’s very important to spend some thought on which subset of models to focus your effort on. Many factors dictate this choice such as the problem statement at hand, the amount of data available, productionization constraints.

While deciding on the right model, the most basic question to consider would be the nature of the problem, whether it is a classification or regression task. This should be clear from the problem statement.

Let us say, we are working on a regression problem. The amount of data available for training would be another important consideration for choosing the model. However, a finer point to consider would be the purpose of the model, that is, whether the model is for prediction or drawing inference.

Prediction

A model needed to predict the output will have to prioritize accuracy over other considerations. Accuracy in this context, of course, means prediction accuracy on unseen data (“test error”) and not on the data model has been trained on (“train error”).

For example, we would want to focus on prediction accuracy if we are predicting stock prices.

In this scenario, the best model would be the one with the best prediction error. Generally, such a model would be more complex and non-linear to capture the variations in the target variable. However, as the model complexity increases, it becomes progressively difficult to interpret all the variations and how each factor affects the output and to what extent.

For example, it is quite easy to visualize data points in a 2-d space whereas it becomes progressively difficult to visualize the points in n-dimensional space as the number of dimensions increase.

Inference

Other than prediction, the model could be useful to approximate the relationship between the target variable and a set of features. For example, it could be useful to understand what factors affect life expectancy such as education, the standard of living. It would be important for policymakers to understand how much would 2 additional years of education improve life expectancy.

When inference is the primary aim of the analysis, it would be preferable to have a more interpretable model over more complex models with better prediction accuracy. For these scenarios, it would be more important to understand the interplay between the factors and how much weight each factor has in the final value for the target variable. Simple, easily explainable models such as linear regression would preferable as a direct look at the coefficients of the linear regression model gives us an idea of the importance of features and the change in the prediction with the change in the value of features.

Have Your Cake and Eat it Too

There could be scenarios where both prediction and inference matter. For example, a model to predict housing prices can not only provide an estimate of the value of the property but also provide an idea of the factors that affect the price of the property and to what extent.

Often in business, a complex model with the best prediction accuracy would need to be paired with some interpretability framework to ease business adoption.

Depending on the problem, if we have an interpretable model with good prediction accuracy, then we can use the model straight away. Otherwise, there are model interpretability techniques such as LIME and SHAP, which could enhance the interpretability of the model.

Issues with Complexity

The model architecture choice depends not only on the use-case but also on others factors such as the amount of data available. A more complex model would need a larger amount of data to provide similar prediction accuracy.

There is a trade-off between model accuracy and model interpretability. As the model becomes more complex, it can better follow the patterns in the data but it also becomes less interpretable. However, if a complex model is trained on limited data, the model can overfit. It would be more “flexible” ie it would fit the variations in the target feature too closely but it is also incorporating noise in the pattern.

Common Machine Learning Models — Interpretability-Accuracy Spectrum (Photo from this paper)

Conclusion

Choosing the right machine learning model depends not only on the business use-case but also on the amount of data available. There might be an over-reliance on prediction accuracy in the industry these days but both prediction and inference offer value in analytics workflows. Both offer different perspectives on understanding the data better and are complementary approaches.

In the future, when starting a new analysis and deciding on model architecture, the end use-case and amount of data available are important factors.

Related Reads