On the process for building software with ML components
By: Christian Kästner
Tags: Machine Learning, Software Process, Machine Learning Pipeline, Software Engineering, Machine Learning Systems

Software engineering education seems to focus a lot on process models (“software development life cycles”) like waterfall, spiral and agile. A number of models also describe the process of developing ML components (“machine learning workflow”). They don’t seem to fit well together, which causes a lot of confusion about the role that software engineering, MLOps and others play in building software systems with ML components, and where the real engineering challenges arise. Far from complete, here is the status of my current thinking… Feedback welcome.

ML Components in Software Systems

Outside of pure science projects, the ML model created with a machine learning framework is rarely the end of a project, but it is a component of a larger system that should be eventually used in production. It may be a smaller component in a larger traditional system, such as integrating the ”design ideas” function into PowerPoint, or building a production system around an innovative ML solution, say building a business like temi.com around a sophisticated speech recognition component. Even in the latter case, where the ML model is certainly the core of the entire application and probably drives 90% of the effort, it still needs to be integrated with may other non-ML components, such as a web frontend, a payment processing system, a data management solution to upload and process audio files, and means to collect telemetry data from an editor for monitoring and later improvement.

Sloppy architectural diagram of a possible online transcription service, with speech recognition as the central ML components and several other non-ML components.

While it is hard to specify what the ML component does (see lengthy discussion elsewhere), it usually has a very clear interface and is easy to encapsulate behind a REST API: Given some input (audio, picture, feature vector, …) return a prediction (class, number, text, …). This component now needs to be integrated with the rest of the system, for example, by calling it on all received audio files in the transcription example.

Let’s talk about process

Process models for software development (software development life cycles) tend to have a bad reputation, but they are useful abstractions to think about how to build a system. Many process models exist but they all tend to go through stages from requirements to design to implementation to quality assurance and they all involve some form of iteration (yes, even the original 1970 waterfall paper is explicit about the need for iteration).

Let’s start with a generic iterative model:

In practice, we do not want to develop the entire system at once, but break it into components that, if we have clear interfaces, can be mostly developed in parallel (for the sake of everybody, I’m skipping a lengthy lecture on information hiding and its limits here).

Figure from Dogru, Ali H., and Murat M. Tanik. “A process model for component-oriented software engineering.” IEEE Software 20, no. 2 (2003): 34–41.

In an ideal setting, we’ll decompose the system into components in an early high-level design step, define interfaces, and then develop those components possibly in parallel.

In practice, nothing will be as clean; we won’t have the perfect decomposition upfront, interfaces won’t remain stable, new components will be added, not everything is done in parallel and there will be iteration within the component development, and teams working on components will probably need to coordinate somewhat.

So, back to ML…

As discussed earlier, ML models are essentially one component of the larger system. In the process above, simply one or more of the components will be an ML component — usually with a fairly obvious interface.

There are a couple of process models specifically for machine learning, such as this from Microsoft: