A Recipe for ML-Ops: Scaling & Operationalising Machine Learning
By: Deepthy Menon
Tags: Renewable Energy, Machine Learning, Predictive Analytics, Data Science, Mlops

Premise

This article is meant to give a conceptual level on what it requires to effectively scale a Machine learning solution.

Unlike traditional s/w where the quality depends on code/platform architecture, Data Science / Machine Learning is all about fine-tuning a metric, and quality depends not just on code/platform architecture, but also data and the extent of tuning. (Source: Databricks )

This is a continuous improvement lifecycle, where one must constantly employ/experiment with multiple techniques, in addition to seamlessly integrating with a robust deployment pipeline.

As a basis for our discussion, this article takes renewable energy, wind forecasting as an example. The ML algorithms used are out of the scope of this article but we’ll see how to put wheels on engines, scaling to cater to terabytes of signals being processed daily.

Setting Objectives

Capturing the problem definition is as important as hiring the best Data Scientists to solve the problem. Taking into account the renewable energy forecasting domain, the single objective can be summarised for a generic audience as the following:

“To have access to the best power forecast available”

Depending on regional regulations/market requirements, objectives are set based on the type of forecasting required, the submission times, frequency of submission and granularity of forecasts (e.g.: hourly, every half hour, every ‘n’ minutes etc).

Data Preparation:

Depending on the problem at hand, the Data Scientists need to decide what kind of data is required to train and deploy models. The Data Engineering teams would work in parallel to create a backbone taking into account the following:

Ingestion method (whether ftp, IOT, APIs, Data Lake etc) Volume/Frequency of Data Cleansing/Cleaning: (Removing noise, duplicates, imputing, etc) Reliability (quality, completeness, addressing data gaps, latency etc)

Model Development/Training/Evaluation:

Data Scientists should not start developing models before examining the nature of data being made available.

The choice algorithms should be tied to the nature of forecasting problem at hand and also how well these respond to irregularities/noise observed in the data.

Platform is another important choice to be made from the beginning:

Scaleability: This aspect covers not only models/code scaleability under various types of forecasts but also condition. Code written for Proofs-Of-Concept seldom scale. The very nature changes while writing code for scalable systems, and even data scientists may need to unlearn and relearn. The platform needs to accommodate for an efficient data pipeline: A robust system to ingest, store and retrieve in time for forecasting multiple times daily. Reproducibility, Tracking & Storage: Initial stage of training for a brand new scenario involves experimentation and a lot more fine-tuning than at later stages. Hence it is important that each experiment be reproducible at all levels: i.e, the conditions in which the model was trained. (Note: Model versioning is different from code version, here the model will be tagged with all the artefacts used to regenerate the same. Frameworks such as MLFlow etc have these features builtin.). Standardisation: Irrespective of the method/algos used, a plug and play format should be employed, and interfaces to the external subsystems (if applicable) should be through standard interfaces ideally encapsulated through APIs. Metrics / KPIs: The choice of performance metric depends on the domain and is mostly driven by the business parameters. For e.g: Accuracy related metrics such as nmae, RMSE, MAPE etc. are being used in the predictive analytics domain. Of course, for the code and input configuration files used to develop the models, principles of S/W Configuration Management & Version Control is implied. Productisation is about adapting the model to the real world conditions under which the model is subjected to. Often on paper or Proofs-Of-Concept usually yield good results at the Cross-Validation level but what is to be thought about in detail is the Operational Accuracy. Usually, it should be a practice to evaluate the models in near field condition before deploying in production. Although model performance is a subjective matter, newer models should be at least a couple of points greater than the ones deployed, regression systems should alert if not and exceptions/outliers logged.

Deployment:

Cost & Sizing: Wind Farms are typically graded by the energy capacity and the number of assets that are under it. Each customer needs a forecast on either on an individual farm basis or as a consolidated submission format with a few farms pooled together, and hence, computing costs could vary drastically for each deployment. At the same time, the choice of platform and architecture should be such that the costs should not go up exponentially as deployments increase, and should flatten out eventually. Scaleable machine learning platforms are constantly evolving, hence frequent upgrades and eol (end-of-life) announcements for libraries are common. Hence any kind of s/w upgrades, be it newer models, or library upgrades should be 100% automated. Again, Solution Architecture plays an important part. Ideally, a container-based implementation is preferred but again, depends on the problem that is being solved and constraints that go with it. Auditability on deployed models is as important as reproducibility; if a job failed or is not performing optimally, it's important to understand why and this feeds directly into the customer response time.

Monitor / Diagnose / Feedback loop:

Machine Learning is a continuous improvement process, so deploying a model means developing monitoring tools delivering actionable insights. These include metrics, outliers, anomalies etc. Most companies underestimate the advantage of building in-house tools for this purpose since these capture the actual working environments these models get exposed to. Models drift over time, due to seasonal changes, training on limited data sets, data pipeline issues etc. Hence it would be in the interest of operationalisation to put a drift monitoring system in place. Pro-active alerting mechanisms ensure that a well-oiled system in place managing a voluminous number of models. At the stage of setting objectives for deployment, the customer may not have baseline data in place to compare performance. But once the daily forecasting comes into place, more insights come which lead to a more refined set of objectives. This last mile connectivity is an important piece which leads to a feedback loop from the field back to product management/data scientists. This often leads to refinement of objectives, retraining, and metrics.

ML-Ops Vs Dev-Ops

Dev-Ops caters to building an automated pipeline producing large scale software solutions. Since the expected o/p is known, a rule-based system fits very well into this paradigm.

However, due to the subjective nature of forecasting solutions (i.e, there is no one-size-fits-all metric here, i.e desired accuracy levels.), hence porting the rigorous rule-based pass/fail approach for pipelines may not be applicable at all levels in ML-Ops.

A typical deployment pipeline could consist of the following:

Airflow, Luigi etc are examples of Workflow authoring tools. These identify dependencies between jobs and have hooks for actions to be taken when there is a failure. Scheduling generally consists of setting up cron jobs — there are many types of syntax, and sometimes when there are complex schedules in place, combine them creatively.

Is a 100% CI/CD pipeline possible for energy forecasting right from training, retraining to deployment? Assuming a very clean data pipeline, theoretically, yes, it is. But most of the time, there will be manual processes involved during training e.g: for new customers, historic data would be smaller and characteristics mostly unknown. So a conscious choice is left to the trainer to choose initial datasets. Likewise, during the scoring process, it is not unusual for accuracy errors to shoot up for a particular day or time of day due to extreme weather conditions — that should not imply the models are not performing.

Further down the line whenever more data becomes available, and as more sophisticated mechanisms evolve, a complete CI/CD can be implemented.

Conclusion