Seven Signs You Might Be Creating ML Technical Debt
By: Moussa Taifi
Tags: Clean Ml Code, Data Sience, Mlops, Machine Learning, Clean Code

Seven Signs You Might Be Creating ML Technical Debt

[7]

The online machine learning ML community agrees that most of the problems that budding ML products face are engineering challenges[1][2]. The leading ML researcher and practitioners seem to confirm that, time and time again, what makes a difference is fast iterations and the ability to add great ML features. The promise is that by investing your resources onto continuous ML delivery practices, significant gains can be achieved from great features and a good deal of iteration; Not from focusing on the latest cutting-edge ML algorithms, techniques, or frameworks.

In this write up we will assume that you are already convinced that the time is right for your business to be building a machine learning powered product. You probably launched your ML service without any machine learning in it. You also probably focused on metrics design and gathering. Besides you already made the decision that you want to use ML instead of the unmaintainable heuristics nightmare that you are trying to replace.

If you are already there, read on — otherwise, checkout how to get beyond phase 0 of ML projects here.

The early signs of ML tech debt

1. You don’t care about infrastructure and are focused instead on launching a complex initial model

[8]

During the initial release, it can be easy to think that most of the problems teams face are ML related. Basically, if your team is discussing things like “Oh, too bad we can’t use the GPT-3 text embeddings, it would really boost the performance of the initial sentiment analysis model.”. This could be a sign of incoming ML tech debt.

If, by focusing on the latest model performance, the team is not clear on how to get samples to the training phase of the models, or there is some ambiguity on what good and bad business outcomes look like, or the team does not know if the model will be applied offline or online, then there should be a screeching alarm with a red neon sign flashing in the teams’ group consciousness.

Besides, on the first projected release, if the team cannot explain how features flow to the system, or the model weights are erratic, or the serving pipeline has a convoluted way of accessing the features necessary for inference, then don’t expect that subsequent releases magically fix these problems.

Finally, if the expectations around the v1 model performance are positively skewed, then the attention to performance will dwarf the attention given to the infrastructure that sets up a more sustainable product-market-fit learning loop.

2. You are mixing testing the ML components with testing the infrastructure

[9]

Testing is a huge eyesore for ML practitioners. Commonly, ML teams report that it is impossible to test the infrastructure without the ML components. If the ML model is inseparable from the rest of the infrastructure, then you might have quite a load of debt to deal with.

Next ML teams sometimes are custom consumers of external data sources. It is quite understandable that nobody else at the company wants the city codes to be one-hot-encoded. This can make it impossible to compare the statistics obtained from the ML-specific transformation, to any external data source.

Finally, another sign is that when a model candidate is selected, after training a model, then there is no validation of the outputs of the model to say if the distribution of the future serving-time scores will match match the offline training-time scores.

The Growth Phase ML Tech Debt Signs

3. You are expecting that the model you are launching will be the only, last, and best one

[10]

Once the ML team finishes the first end to end system, the “technical cash-flow problems” can start showing up. The ML teams usually find themselves looking at the accumulated debt from the initial phase, where the focus was on getting the features to the models, collecting metrics by cutting corners, and propping up the serving infrastructure on various kinds of stilts.

One misconception repeatedly reported in the ML Eng industry is that the first model that teams are launching will be the only, last, and best one. Some might even believe that there will ever be a time when models will stop being launched. This sort of attitude, one-and-done, is usually rectified rapidly as the team discovers all the features that are still to be included in the model. While daily retraining a no-change ML pipeline with new data is becoming standard, at the very minimum, expecting a monthly or quarterly release for 2 years of an improved model pipeline should be the de-facto baseline for ML teams.

This phase is an exciting time for ML teams; they start focusing on improving the model with new features, tuning hyper-parameters, creating new old-feature combinations, and even rethinking the proxy business metrics they are optimizing. Unfortunately, this is where most technical debt gets created because the project has been running for a while, so even more maintainability corners are cut to release new improvements on time.

Signs that the ML pipeline needs some care and attention is for the team to ask itself:

How easy would it be to remove features from the existing model?

How easy would it be to add new features or combine them with existing ones?

What would it take to have copies of the whole ML pipeline? How hard would it be to have dark launched pipelines to release and compare their performance to the mainline?

Does the team prioritize including costly features into the model rather than frequently releasing incremental versions and learning what works and what does not from live user traffic?

If an ML team finds itself locked-in and unable to move because of the previous design and architectural decisions, that should be a warning sign of growing technical debt in the system.

4. You leave unused features in your pipeline long after they stopped being used

[11]

The most effective way of increasing performance, great features, can also be the most insidious technical debt creators. If you hear your team saying that we should keep an unused feature in the feature transformation jobs “just in case” then it can be a sign that the ML tech debt is growing.

This sign goes hand in hand with situations where adding a new encouraging feature can’t be used without having to wrestle with the code because an unused feature is conflicting with the new feature. This sign points to the rigidity of the design of the pipeline where the feature transformation are probably overly intertwined with the model training/serving code.

Finally, sometimes even if there is a clean way to removing unused features, there is no easy way to distinguish the usefulness of a feature because there is no process in place to rank the utility of features based, for example, on their coverage and correlation with the labels.

5. You are looking for ways to schedule your notebooks as production pipelines

[12]

Jupyter notebooks are amazing tools for exploring and prototyping your ML pipelines. The community is fantastic and in the future there might be ways to build solid software based on notebooks, but experts are slowly speaking against attempting to use notebooks in production as is.

Scheduling notebooks and using them in production might happen in the future as the technology underlying the notebooks evolve. However, as of 2020[5], the guidance is to extract the smarts out of notebooks and use standard integrated development environments that promote cleaner architectures, modularization, automated testing, and standard software practices.

For now if you are attempting, or are currently scheduling notebooks as prod, it might work for a while but your productivity will hit a wall due to the inherent mixing of concerns and technical debt that accumulates in notebooks over time.

The Training-Serving ML Debt Trap

6. You use two separate implementations between the training and serving pipelines

[13]

There is an easy one to spot and a hard one to detect for this sign.

The easy one to spot is deciding to use a different programming language or even a different implementation of the ML code between the training and serving/scoring pipeline. This can be a clear sign that you are giving up on any sort of iteration and reuse after the first release. Getting a paratrooper engineering squad to re-implement your classification scoring pipeline from scratch is a recipe for disaster. From an software perspective the training code will dramatically drift away from the scoring code due to their nearly opposite concerns. Training is backward facing and errors might trigger an internal email alert, while scoring will most probably be client facing which is going to light a fire in your Slack channels at ungodly hours.

The second sign occurs specifically when scoring is an online processing operation. While the team might have over optimized the code for batch processing during the training and model selection phases, the online processing requires the system to handle each request as it hits the prediction service. If you don’t have a plan on how to bridge the difference in the nature of the data processing that the online and the offline systems deal with, then this can also be a sign of potential growing ML tech debt that might come to haunt you later.

7. You are not totally sure 100% what CI/CD stands for

CD Continuous Delivery and CD4ML Continuous Delivery for Machine Learning.

[14]

This is related to point #3. Plan to iterate.

Continuous delivery[3] is an SW practice that is extremely applicable to the ML engineering world. There are many signs that the CD practice would help you evolve your ML product. If you are rarely releasing models in production and multiple “improvements” keep piling up waiting for the Q4 release train[6], then the fear of the impact of the new major release keeps growing. The longer a change is stuck in the release train, the more uncertainty there is about whether it will work in prod.

Another sign that CD is needed is when there is a costly process to release models to production. If the project managers are budgeting for a change window to allow for all the teams involved in the data pipelines to sync up, then this fixed cost, attributable to the release process, should be taken as a sign.

A lack of a solid deployment pipeline contributes to probably unnecessary coordination, but it can reduce the happiness of the team members. Their model improvements linger unused, un-deployed, and might remove that vital feedback loop between your ML engineers and the end-users where most of the sense of responsibility, continuous learning, and usefulness comes from. Similarly, if there is no way to organize the code and the deployment pipeline to allow for frequent A/B testing, then it might be a sign that 2/3 of the functionality built is not used by users[3].

8. Bonus point: You realize that ML Eng is a lot like Software Eng but you never found a good book bridging these two worlds

Machine Learning (ML) pipelines are software pipelines after all.

I had the unique chance to research this topic, as a quarantine project, in order to distill, and organize what I learned over the years, in this book:

Clean Machine Learning Code.

This book will help the readers apply practical software engineering principles to prevent failures in their machine learning software craftsmanship journey.

Get yourself a copy of my latest book today!

Thanks for the support.