Who moved my binaries?
By: Christian Melchiorre
Tags: DevOps, Machine Learning, Software Development

We have now reached our third post in this series of articles about the experiments I carried out in adapting my software development pipeline to include my new ML stuff. To summarize the previous parts, we discussed about source code control and versioning and about the build process aimed at producing the binary bundles that make up our application. In particular, we have seen how wanting to exploit Tensorflow and Keras libraries, I was forced to include parts of code in a given programming language (eg. Python) in what previuosly was an application principally consisting in modules written for a different technology stack (in my case, Java).

What is missing so far is what to do with the different pieces of binary artefacts that we get as a consequence of this write-and-build process. This phase too is affected by the polyglot nature of our application (that is, the fact that different components are written in different programming language), since what results from it consists in a number of binary packages in different, incompatible formats: .jar files for java modules, .whl (“wheels”) for python, and so on and so forth.

We’ll need to take care of this. But first, let’s first try to better understand what we’re dealing with, since many different things can fall under the generic definition of “binary package”, often serving different puposes and used in different stages of our software development life cycle. Just to make some examples, we might have:

Reusable software libraries, providing well documented and encapsulated functionality. These are imported and embedded in our code as dependencies (Remember? We chose tools like Maven to take care of these in the first place). This should be familiar to everybody: we’re talking about stuff like Tensorflow or Keras, which are built on top of lower level APIs (eg. Numpy) to provide a more general level of abstraction. Of course, here we are not considering general purpose libraries publicly available, like the two above, but the ones which are designed and produced by our team to meet the specific needs and requirements of our company or of our application. An example? Don’t know… maybe your company’s data are a bunch of image files that need very specific preprocessing steps. Instead of having to repeat code in each application component to perform the preproc, we abstract and bundle this functionality in a reusable library, with its own API, and keep it available in our repository.

Application components or modules. Well, actually this too is a very generic term, and technically there’s not a clear-cut difference between these and the above reusable libraries. I just like to use different terminology to distinguish the former case, where the functionality is developed with future reuse in mind, from any software module developed to run as a very specific part of an application, maybe conforming to some given component model (think an EJB providing some specific business functionality in a e-commerce application, a Flask python REST service, etc.).

Ready-to-deploy executables. When all it’s said and done, what we need is some piece of software to run somewhere. In a classic (so-called “monolithic”) application this could be a single executable file (and .exe for windows systems, an executable .jar, or something like a .war/.ear archive in the case of Java enterprise apps), bundling together all our libraries, application components, and so on. One such bundle is ready to be configured, deployed and run on some target environment. In microservices architectures[i], on the other hand, our application can be composed several, independent deployable and executable modules.

Now comes our guest star: Pretrained ML models. With this, I refer to a serialized copy of all the information needed to recreate in memory a previously trained model without for instance having to re-train it from scratch. This information could consist in just a couple of numeric parameters for a simple logistic regression model, up to matrices of millions of floating point values for complex multi-layer deep neural networks.

You could be tempted to conceptually consider these as out of context as concerns the current discussion — that is, to view them just as plain data files. After all, they do not directly provide exectuable functionality in themselves, but need some runtime component or serving environment to bring them to life. They might even be stored in a non-binary (eg. text) format[ii], putting them definitely outside the definition of “software binaries”.

As you’d guess, I prefer to adopt the opposite view, treatig them as just another type of module providing executable functionality in some serialized form — and including them in this discussion.

Let’s now talk about how to manage all the different kinds of binaries we’ve discussed so far. First of all, these exist for different purposes, so different tools will come in handy to manage them. Roughly speaking, we can group the above binaries into two groups, according to their intended usage.

A first group is composed by those binaries that we need in our code development process as dependencies — pieces of software we bundle in our application code to build other binaries from. Not only binaries generated by building our own code, but also third-party binary components, such as free open source or licensed commercial libraries. What we need in this case is a binary repository manager — a dedicated server application designed to store, version and provide binary components. The choice will typically depend on the package format/build tool we’re using. For Maven-based projects, for instance, one of such tools I’ve used in the past is the Sonatype Nexus[iii] repository, but many valid alternatives exist, one notable example is JFrog’s Artifactory[iv]. Artifactory is, too, mainly Maven-oriented, but it is advertised as “the only Universal Repository Manager supporting all major packaging formats, build tools and CI servers” — that includes standard Python tools, such as pip and distutils (though I guess that if you’re working exclusively with Python you could get away with just installing your private PyPI server on your CI env).

As for the second group , once youìve built your application or service, you need to keep the finished “executable” bundles ready to be deployed and configured in some target runtime environment. Again, the use of different technologies could make things complicated, but here is where a nice little tool named docker comes to our rescue.

It’s unlikely that you haven’t heard about docker yet. This technology emerged as a sort of “revolution” a few years ago, and I sincerely can no longer imagine how I could live without it before. It provides a way to uniformly package applications in so-called “containers”, a sort of lightweight self-contained virtual machines, bundling application code with all its dependencies as a fully self-contained highly portable package. [v]

Docker comes with its own repository, named the docker registry, that you can install and secure on your own servers to store and manage the your containers’ images — besidese this, it seems that Artifactory is also compatible with the docker format, and can be used as a repository for container images.

What about serialized pretrained ML models? As I said earlier, these are usually a bundle of parameters stored in some serialized format: TensorFlow stores files in Google’s Protocol Buffer format, Keras provides a basic save format using the HDF5 standard, or you can just custom-pickle your numpy arrays into your own data structure.

The fact is, while in the simplest cases these files could consist just in a limited bunch of numbers (as in a simple linear regression model), in general they can turn into huge files with millions of floating point parameters — for instance in the case of very deep neural networks. Managing huge files means facing big limitations in storing and, notably, versioning. Luckily, there is a GIT extension specifically addressing the issue of large file management[vi]. An even better piece of news is that it seems that Artifactory supports GitLFS, too. (Again, I’ve never worked with Artifactory as a binary repository yet, but it is definitely on my to do list…).

More specifically, recent developments have seen the appearance of a whole new class of products that falls under the name model servers — examples are TensorFlow Serving, Apache MMS (Model Server for Apache MXNet) , clipper.ai and others. With respect to our present discussion, these seems more oriented to the phase of deployment to production — rather than the storage and versioning of models to support the software development stage — so, maybe these are worth their own dedicated discussion in a future post.

I’ve almost reached the end of this rather long article. Let’s wrap up what we have so far:

We have set up a number of software pieces of “machinery” in our software factory, a source code versioning server (gitlab), a binary repository (nexus, artifactury, …), some model server and a repository for deployable docker containers.

What’s still missing is some “glue” that keeps everything together. With this I mean something that allows to go from writing code to building to testing to deploying to … whatever, without having to manually interact with each of the software tools we’ve talked about: namely, we need some kind of Continuous Integration engine (like Jenkins), that helps to automate the whole process. But… I think that this post has already become too long, so I’ll leave this subject too for one of the next times.

Hope you fund the stuff we discussed interesting!

Links and references

[i] Check my previous post

[ii] For instance, they could be stored as an XML-based interchange format, like in the case of PMML, which is the de facto standard to represent predictive solutions.

[iii] https://www.sonatype.com/nexus-repository-sonatype

[iv] https://jfrog.com/artifactory/

[v] https://www.docker.com/

[vi] https://git-lfs.github.com/