10 Pitfalls and 11 Best-Practices for the Design Phase of a Machine Learning Application Project
By: Bruce H. Cottman
Tags: Cloud Computing, Data Science, DevOps, Best Practices, Machine Learning

We concentrate on pitfalls you may encounter in the beginning design phase of a Machine Learning project. We detail our best-practices to avoid these pitfalls.

Figure 1. Iterative Lifecycle of a Machine Learning Application (MLA).

Have you successfully designed, trained, and tested a machine learning application (MLA)? Despite a vetting in the lab, did the MLA not behave acceptably, maybe even fail in production?

If so, read on, and I shall detail, pitfalls my colleagues and I encountered. I then detail best-practices, some of which are solutions we developed to avoid these pitfalls.

Our Approach

Lowering the high cost and burden of machine learning application design, development, and deployment is the methodology domain of DataOps, DevOps, MLOps, GitOps, CloudOps… xOps, where Ops stands for Operations [1].

We believe that you should learn how to crawl, then walk, then run and then maybe fly, in that order.

We think xOps is the walking, running, and flying in the MLA lifecycle.

We start with crawling.

We focus primarily on the pitfalls and best-practices at the launch of a project — a machine learning application (MLA) design phase.

Project management is described as pitfalls and best practices.

Project management can be laid out as a graph (edges and nodes) of steps. Tasks that are sequentially accomplished in a specific order is the Waterfall project management methodology. An example would be the sequential ordering of design, development, test, and finally, deployment.

Note: We have had two project successes and many more project failures, doing these tasks sequentially with the waterfall project methodology. For the failed projects, we either ran out of time or money. All projects were painful and required burning up many of our weekends.

MLA Project management is imagined as a continuous cyclical graph of tasks (see Figure1). We also think of an application project as a checklist of tasks, where each task is partially completed. Some of these tasks can be performed in parallel, some are dependent on other tasks, and some are repeated until the application is dust bin.

Note: We suggest that you accomplish tasks iteratively and asynchronously,

Hopefully, the above is the entirety of what we need to discuss about MLA project management methodology. From now on, we focus on some of the more frequent pitfalls that we encountered in the MLA design phase. These pitfalls were derived from about four decades of combined years of experience and about eighteen MLA projects.

For any pitfall listed, one or more best-practice workarounds or solutions are detailed. Each best-practice is usually a component task, which has few prior inputs as possible. You will notice that some tasks have unavoidable starting dependencies.

We also fell into development and production pitfalls further down the MLA project lifecycle. We dug ourselves out (usually) with best-practices.

Pitfalls and Best Practices

Pitfall 1: A machine learning solution looking for a problem.

We put this pitfall at position #1 because, as a consulting group, mostly in 2016 and earlier, we often put in a machine learning solution when the business problem did not require it.

What was expected of all these clients is that they wanted to say they were using Machine Learning (ML) applications, whether they needed ML or were not ready for ML.

Pitfall 2: No Data, No Problem Identified

Do not laugh, but in the early days before 2017, we were engaged by the client with no data identified or no problem identified.

We have found that both Pitfall #1 and Pitfall #2 need resolution before proceeding.

Best Practice 1: Understand the Business Problem, Define the MLA Solution.

Do not engage with firms that do not have or will not let you define a viable machine learning solution.

Best Practice 2: Low Cost Means Low Hanging Fruit.

Your first projects should add capabilities, not replace and improve existing capabilities.

Until you have proven the value-add of machine learning, avoid adding machine learning applications to an organization’s legacy (existing) system. Easier said than done!

Note: You may end up understanding an organizational or department business process and the system implementation of that process. The system implementation will have little or no insight into the business case you want to solve using machine learning.

Best Practice 3: Find and identify business case, process flow, or /and dataflow diagrams.

KPIs (Key Performance Indicators) are a great indicator of what the organization thinks is important. The organization may measure an assortment of department, process, and project health metrics.

Also, poll the stakeholders and find out what frustrates them the most.

For example, a hotel chain measures room availability, or a restaurant measures table availability. ML can predict future booking based on past booking. However, more valuable ML is decreasing availability.

Best Practice 4: Work on one MLA at a Time.

Once you identify a business case(s) problem, propose one MLA solution. You can launch other MLA projects after the first MLA is a measurable success.

Best Practice 5: When the first MLA is successful, increase investment by launching more MLA projects.

This best practice is prominent. What is not so obvious is that your sponsors may want to start more MLA projects before the first MLA project is rolled out to production.

Your sponsor (or you) may be very excited by the beautiful results on train, test, and validation in the lab. Warn yourself and them about rollout disasters of the past [2].

Pitfall 3. Design a Monolithic Architecture.

Creating monolithic applications causes the waterfall development process. Different steps and parts of an application are very dependent on each other.

Best Practice 6: Design a component pipeline or micro-services architecture.

Design your MLA as a pipeline of components or microservices. In this case, each component is a separable machine learning microservice. Each component is entirely executable independent of each other, requiring well-defined inputs and outputs.

Tip: A component-based architecture allows your team to develop different parts of the MLA asynchronously.

Best Practice 7: Design an MLA blueprint.

The first part of your design specifies the functional and non-functional requirements of your MLA. The requirements identified flesh out the architecture design of your MLA. Also, requirements will dictate production tactics and should give specific testing of the MLA.

Tip: Develop and share widely MLA architecture blueprints that show the end-to-end integration of each logical component in the MLA pipeline.

Best Practice 8: Use your Software Engineering Best-Practices.

Apply your software development project experience to your machine learning application.

ML is different as incoming data and tuning parameters disproportionately affect a stochastic MLA more than a deterministic application. However, you will find that most of your past software engineering methodology and best-practices apply to your ML projects.

Pitfall 4. Re-invent the Wheel.

We have seen many examples of projects that had to be restarted because the research of prior work was not done.

Good artists borrow, great artists steal — Pablo Ruiz Picasso (25 October 1881–8 April 1973)

Tip: Spend some time googling on GitHub at the beginning and then continue through the project. Tip: Our experience that good papers on arvix.org are implemented in code and placed at GitHub. Code snippets help launch your project.

Pitfall 5: Too Simple a Solution.

For smaller-scale projects, an out-of-the-box solution may be enough. Unfortunately, custom functionality or multiple ML models in tandem may be necessary to ensure a lead on the competition.

Best Practice 9: A Natural Language Processing (NLP) Application Architecture Requires a Transformer.

We decided to include an outlier best practice in this blog because there has been a significant shift in NLP applications in the past two years.

We had deployed a GLOVE-based NLP application in 2017[ 3]. The BERT transformer appeared in 2018 [4]. BERT resulted in a large jump in NLP performance with two key improvements:

BERT was pre-trained on a corpus size at least four orders of magnitude (10,000x) more immense than our GLOVE-based ML model; BERT had a model parameter count of about two orders of magnitude (100x) more immense than our GLOVE-based ML model.

“[…] scaling to extreme model sizes also leads to large improvements […]” (Devlin et al., 2018)

Pitfall 6. Do not Communicate Progress

No matter how successful the MLA, there will be project hiccups along the way. We have found a 1–2 page weekly status report to the client project sponsor and manager, even though they did not ask for it, eliminating most miscommunication problems in the project.

Battle story: Once, we were fired, but luckily, we got rehired two days later when several misunderstandings were cleared up. Ever since weekly status reports are a required best practice.

Pitfall 7. You got the Business Case wrong.

No matter where you are in the project, your requirements are wrong, and you need to stop and reassess. Be prepared to start over or be fired. Document your new proposed solution. Life is too short to continue on and rollout an MLA solution that needs a problem.

Pitfall 8. ML app does not satisfy the need.

Pitfall 8 is similar to Pitfall 7. We list it here because you got the business case right, but the business need is now of low priority or went away. The only thing you can do in this case is proposed to answer a different need. Be prepared to walk away.

Pitfall 9. The client has no Previous Experience or Knowledge about ML.

In earlier times, we held classes on “How to benefit from Machine Learning?’. Now we have we point them to Awesome-X URLs on Github, Coursera, and Medium blogs.

We have not thrown out the slide-deck as we still use it time-to-time.

Tip: Mentor the client’s development staff in ML practices.

Pitfall 10. No ML deployment infrastructure

We have only one answer (viable solution) to “We do not have ML deployment infrastructure.” — get a cloud account.

Rolling out your own infrastructure can be done. Apple, Amazon, Goggle, IBM, Zillow, and other multi-billion (or trillion) companies have done it. Our answer — get a cloud account.

Best Practice 10: Deployment in the Cloud

Google Cloud Platform (GCP) is one of the primary options for cloud-based deployment of ML models. Other choices are AWS, Microsoft Azure, Paperspace, etc.

Tip: Our favorite is the free GCP Colab [5].

Best Practice 11: Periodic Audits of Solution

Machine learning state-of-the-art (SOTA) change is happening and speeding up. You don’t want to be blind-sided by another BERT-type occurrence.

Keep up with the literature. You may also evangelize your new great ML projects with your clients.