3 Data Foundation Problems That You Should Fix In 2021
By: Pathairush Seeda
Tags: Data Analytics, Data Quality, Machine Learning, Data Science, Data Governance

ML Deployment

Photo by Vadim Sherbakov on Unsplash

The usual product of data science is a predictive model. When starting the journey three years ago, I delivered the prediction model's result in a CSV file format. And the model is saved in a pickle format for manually scoring by a data scientist.

That’s a traditional way to get things done.

Luckily, the technology has developed so fast that today we have many libraries/frameworks like MLflow or others for deploying machine learning models.

At my previous company, the engineering team also developed the platform for automating predictive model scoring. They combined the latest framework mentioned above with the in-house knowledge to create a unique platform.

It makes a data scientist's life easier. It contains several useful features and reduces the step for data scientists to deploy the model.

It takes care of both feature store and predictive model’s metrics. But it’s still has a gap to improve. Here they are.

Monitoring

The deployed model needs to be monitored. The performance of the model can change for many reasons. A good blog from Databricks described how many problems you can face after deploying the model. I highly recommend you to read it.

In summary, there are three issues that can make the prediction looks weird after model deployment

1) Data diff, 2) Concept diff, and 3) Upstream data changes.

Those problems can silently affect the prediction result of the model. We should plan to prevent it because our model will be used for something valuable like recommending the product to the end customer.

If the model goes wrong, it can make a huge loss to our business.

The consistency of the prediction result is also important. For monitoring it, there are two parts to be concerned here

1) Data, and 2) Model

We need to ensure the quality by implementing the data quality checking for the data side as in the previous section. The system should alarm the data quality defect first before you know the deviation/defect from the model prediction.

Also, we can use a library like MLflow to track the model metrics such as AUC, Precision, Recall, or others for the model part. Then, we can set the threshold to trigger the alarm if those metrics deviate from the current standard you set.

You can trackback the reason behind the changes with the help of a data quality checking system.

Finally, a dashboard or report can be created from logging metrics. It helps tracking the metrics like the number of models that meet the standard performance or the changes in prediction distribution from the model.

Re-training model

Here is another problem from my experience. When you use the model for some period, the longer time the model is used, the more likely it becomes stale.

You need to have a re-training feature for the ML deployment system. Otherwise, there will be a lot of hard work waiting for data scientist at the end of the ML pipeline.

The re-training model is to add more new data or shift the period of data you use to make the model adapt to the customer's current behavior.

Pro tips : It’s hard to define when to re-training the model. You can make a simulation of the several re-training periods and see when the model performance drops.

Testing

The last problem about the ML deployment is testing process. Testing usually is a must thing to do in software development. We expect the system to reproduce the same result every time.

But it’s hard to do in ML deployment because the model is figuring out the information from the uncertainty in data, it’s difficult to make a reproduceable result after you fitting the model.

We can do our best by testing other aspects of the deployment.

For example, the type of input data and output prediction, the unexpected behavior in results such as null value or the prediction that’s doesn’t make sense.

We can generate the test case for each category of the model like classification, regression.

This helps improving the model output quality and lowers the chance of error for the consumer side system (downstream part) like the front end to display its prediction value.