Evaluate your model properly
By: Jonathan Choi
Tags: Machine Learning, Business, Data Science

Evaluate your model properly

It’s not all about the statistics

The progress we are seeing in machine learning is undeniable, in any given week, we see new algorithms being researched and theorised, new libraries being released to the open-source community and new accuracy benchmarks being eclipsed. There is no doubt that the data science community are producing tools that are getting better at optimisation towards the objective. But that’s not what I want to talk about today. As we are getting better at hitting the target, I want to question whether the target is the right one all along. As model fitting gets more and more complicated, model evaluation seems to have stayed the same. It seems to make sense because there are only so many metrics you can use to assess a model’s predictions, but actually, there is more to model evaluation than statistical errors and biases.

Let’s take a look at a real-life example. When we are watching a race, we measure the car performance by lap time, which is a function of speed and handling amongst other more detailed metrics. When we are buying a car, how fast it can go is probably not the most important thing in our evaluation criteria. When we are thinking about investing in a car company, how fast their cars can go is almost irrelevant. The simple metric of speed is crucial in the well-defined environment of racing, but its importance diminishes as we move into a more general context. This simple example shows us that our evaluation criteria change significantly based on the context we are in and we have all experienced it before. However, this simple logic seems to be obscured by layers of technical concepts when it comes to evaluation of machine learning models and data science projects in general.

Photo by William Warby on Unsplash

The current approach that I have witnessed for too often involves trying to explain to a business executive why improving the root mean squared error by a couple of percentage points matters. That’s akin to us trying to decide whether we should invest in a car manufacturer because their new car has a slight improvement in top speed if we continue with the previous example. It’s no wonder that conversations about technical metric improvement never really get through to senior stakeholders. I think it’s worth noting that I am not saying that technical metrics don’t matter, they absolutely do, but as we move from a technical solution to a business solution, we need to measure the right metrics accordingly.

Measure what matters

By now, we should have learnt that asking the right questions and finding the right problem to solve is important when it comes to doing data science productively. However, we often don’t follow up on the other side when it comes to evaluation. For example, if we want to evaluate whether a churn prediction model can help with customer retention to increase recurring revenue. Knowing the likelihood of churn of a customer is only one part of the equation. We also need to consider other factors like how likely we can convince that customer not to churn, how much would that cost us to do so and what resources we have available, etc. Ultimately, we should be evaluating our model based on predicted revenue retained because that’s what we care about. Using the right metric will ensure that we are not wasting time on optimising things that don’t matter.

This is often neglected because unlike modelling algorithms where we can just install a library and use, there are no standard metrics that will be perfectly applicable to our specific situation out of the box. As analytics professionals, this is where we need to spend time thinking about what is the right evaluation criteria because it’s our responsibility to ensure that our solution is effective when put into practice.

Start with the end in mind

Another common pitfall that I have seen quite often is that we jump into the analysis without establishing how we are going to evaluate the output. No doubt it is exciting to jump straight into the project but if we don’t define the evaluation criteria upfront, we will likely be biased towards the solution that we have built as we are committed to it. Designing the evaluation process before we start building a solution and standardising the evaluation process can ensure that we evaluate different ideas objectively.

The paradox of choice

How many times have we found ourselves lost in the sea of metrics when it comes to model diagnostics. Take a simple classification problem, we can evaluate the model using the confusion matrix, ROC curve and F1 score just to name a few. As we have discussed earlier, none of these metrics should be the final one that we use to evaluate the model because they don’t tell the full pictures, however that doesn’t mean that they are not helpful. These diagnostic metrics enable us to understand what the model is doing and they help us identify the area of improvements that we can iterate on. It’s important to understand what metric is useful for what scenario and understand the relationships between them.

Find the right business metric

Data science is the process of using data to solve business problems. We understood the importance of translating that business problem to a technical one that can be solved with data, we also need to understand that we need to translate the results back to a business one that the wider company will understand. This is a process of modelling the business and understanding the relationships between different business units and what metric is important to them. While this seems like a less ‘technical’ exercise, this is a crucial part of turning a well-performing model into a useful solution. This is also where domain expertise comes in because that will help translate a technical metric to a business metric.

Back to basics

Unlike algorithm development, model evaluation is not about complexity and finding a better way to optimise. Model evaluation is about simplicity and finding the right representation of performance. If a good machine learning model is a fast car, then a good model evaluation is a good navigation system. If we want to get to the right destination, we need to focus on both. So next time, when you are building a solution, go back to basics and think about what metrics matter and how you can use that to evaluate your approach.