Rethinking the AI / Machine Learning Lifecycle for the Enterprise
By: Eric Broda
Tags: Artificial Intelligence, Machine Learning, Data Science, AI, DevOps

Why are these requirements important? In financial services, reproducibility, traceability, and verifiability are explicitly regulated (ie: in the European Union, United States, and Canada) and cannot be overlooked. But similar requirements are also found in many other industries from health care and biotech to government security. In fact, now even enterprises in modestly regulated industries are finding that the benefits of reproducibility, traceability, and verifiability far outweigh their costs.

In many ways this is not surprising: How much quicker and more frequently could models be updated if data scientists could easily modify a model that was last worked on a few months ago without scrambling to find model source code and the correct training files? How much easier and quicker could auditor and regulator questions be answered if data scientists could confirm training data lineage and consistently re-create the training files from raw source? How much faster could models be promoted into production if data scientists could easily find the lifecycle outputs required by an enterprise gating process?

Most data scientists would overwhelmingly agree that the benefits of reproducibility, traceability, and verifiability improve their effectiveness and agility.

Reproducibility, Traceability, and Verifiability Drive Data Science Lifecycle Maturity

Consider a few common scenarios that data scientists experience regularly, and the shortcomings in current data science lifecycles:

A data scientist must address production data drift which is leading to degraded model performance: An enterprise with a mature lifecycle can quickly re-establish the model baseline by acquiring the model source code, and training files and then reproduce results, after which a model can be updated and re-trained. Unfortunately, in many enterprises this simple scenario is difficult to achieve since linkages between model source code and training data are not maintained. A data scientist must prove to an auditor or regulator that a model achieves a correct result based upon original business events: In this case the data scientist in a mature enterprise would be able to demonstrate how raw data files are correctly enriched, transformed, and/or aggregated such that the model can be verifiably trained based upon a traceable lineage of data. However, most enterprises are unable to link training data lineage and the associated transformations that must come together to form a verifiable model. A data scientist must demonstrate that all enterprise gates and checks have been successfully completed prior to promoting a model: To facilitate this outcome in a mature enterprise, DEVOPS and MLOPS processes would automatically capture critical artifacts and required AI/ML lifecycle outputs required by an enterprise gating process. Unfortunately, few enterprises automatically (or even manually, in some unfortunate cases) capture and retain critical outputs from the AI/ML lifecycle required to allow easy and quick promotion of a model.

Clearly there are significant opportunities to improve an enterprise’s data science lifecycle maturity by focussing on reproducibility, traceability, and verifiability. The real question is not if an enterprise should address the issues, rather the question is how quickly an enterprise must address these issues.

The New Data Science Lifecycle Enabler: the Lifecycle Catalog

Reproducibility, traceability, and verifiability are enabled by several simple capabilities: capturing artifacts about models as they evolve over the data science lifecycle, storing these artifacts, and, searching and viewing artifacts. These capabilities are enabled with slight additions to the current data science lifecycle while providing a tool — a “Lifecycle Catalog” — that provides a view port into the data science lifecycle.