The Data Science Research-Production Chasm
By: Joe Plattenburg
Tags: Ci Cd Pipeline, Ml Pipeline, Data Science, Data Engineering

Tell me if this situation sounds familiar: You’ve shown in a proof-of-concept setting that your new ML model will be effective, but after a full-scale fit, the performance isn’t quite what you expected. Or maybe after applying feature transformations in a production environment, the resulting features are close, but not exactly what they were in your research environment. Or maybe you’ve successfully validated model predictions on historical data, but when that model is pushed to production, there are suddenly a handful of predictions that differ from the research environment. It should be the same model artifact and basically the same code pipeline! What’s going on here?

Welcome to the chasm between data science research and production.

Now, you may be thinking that these problems already have solutions, like:

Version-controlled code, e.g., Git

ML pipelining framework for defining your process, e.g., Airflow

Cloud storage for critical modeling files, e.g., S3

Team-wide conventions and data integrity tests

But these tools are only as good as how they are implemented, and there are still a host of opportunities for breakdowns within components and at their interfaces. But maybe “that sales rep from SaaS company XYZ showed me last week how their platform solves all of these problems!” (Spoiler alert: It probably won’t, but they’ll be happy to take your money for the privilege.) Unfortunately, the insidious ways that tooling and process failures magnify small discrepancies between research and production are all too common — and often turn cracks into chasms.

To some extent, this is the nature of research. We don’t know a priori if something will work or not, and ideas need to be tested as quickly as possible to get them to market. No one would cast blame for taking judicious shortcuts — after all, this is the idea behind the well-known concept of technical debt. The important part, however, is that this debt eventually gets paid down. In software engineering, ongoing technical debt can result in code that is hard to test and maintain. Here we discuss ways in which data science technical debt can manifest as similar but distinct problems, and how to address them early on.

A motivating example

Let’s take an example from the insurance industry: Suppose a data scientist wants to build a new model to predict insurance losses using a new feature from an external source, and then determine where that model shows an improvement over the existing one.

The existing pipeline transforms several different datasets (e.g. maybe braking events, distracted driving events, etc.) into a “user-aggregated” dataset (X) via a job (f), which feeds into a model (g) that predicts probability of auto accident (y) for that user.

Figure 1: The current state of the world

The new data source (Xj) has information about the vehicle in which the trips were taken, delivered via API requests, and the researcher hopes that this vehicle information may help to improve their hard-braking feature. Therefore, they need that feature available as an input to f. In a perfect world, the new pipeline would look like this:

Figure 2: The ideal future state of the world

However, because this feature comes from a new source, the data engineering team informs the researcher that it will be several weeks before they can modify the ETL pipeline to integrate the new feature via the partner’s API. In addition, the feature can only be integrated once a contract has been negotiated with the external provider; as such, the finance team wants a cost-benefit analysis detailing the model improvement to justify the cost. But of course, the value component of this analysis cannot be generated until the feature can be integrated into the ETL pipeline.

To short-circuit this Catch-22, the enterprising data scientist secures access to a modestly sized “evaluation dataset” (Xj’’) with which they seek to assess the model improvement (e.g. how much better y’ is than y). However, this dataset does not come directly from the production API — maybe it was manually scraped from the underlying dataset. Therefore, the researcher comes up with some custom research code in a notebook that pulls in some data from the production environment, and then joins it onto the new evaluation dataset to produce X’’, an estimate of what X’ might eventually be.

Figure 3: The realistic research state of the world

With the evaluation dataset in hand, the researcher uses the combined data to fit a new model, g’’, and soon discovers that the results look great! The improved features significantly increase the accuracy of the model. Obviously the business wants to get this new model into production as quickly as possible, so the API integration gets fast-tracked. There is still quite a bit of effort involved in trying to translate the custom code from the custom analysis script (h’’) into the ETL job (f’), but finally, after several weeks and a couple delays (the stakeholders have already widely disseminated the new-and-improved results in the most recent investor meeting, but no pressure…), a production-ready model is complete. Just before releasing it into the wild, the data scientist performs one last check between the production predictions (y’) and the research results (y’’).

Uh oh. They don’t match up. Why don’t they match up?! In the best case scenario, the performance metrics are “about the same” and the company needs to get the model out the door, so it is just shipped with some trepidation about the discrepancy. In the worst case, the performance metrics have significantly deteriorated and the researcher descends into a weeks-long purgatory of trying to track down bugs and code differences.