A Reactive Machine Learning system on top of Kubernetes
By: Manuel Lara Caro
Tags: Machine Learning, Microservices, Apache Spark, Kubernetes, DevOps

In order to fulfil his mission, Klanus started to study the history analysing all the famous Jedi, all the infamous Jedi that fell to the dark side and the fearsome Sith Lords.

As part of his analysis, Klain gathered some information from all of them in order to find some patterns that could allow him to predict if a new padawan could fall to the dark side.

The specific information that Klain collected was:

Name , just for to have a reference.

, just for to have a reference. Midichlorian , number that can be use to measures sensitivity to the Force.

, number that can be use to measures sensitivity to the Force. Species

Gender

Homeworld , planet of birth

, planet of birth Side

So, with that information in place, Klanus created a sophisticated algorithm that took as input the information gathered to learn a mathematical model that was capable to make predictions.

Over time, Klanus and the rest of Jedi Masters could check that the algorithm worked really well, so the Jedi High Council decided to use it at scale. If the algorithm was capable to predict which padawans would become a Sith, then it also should predict which people could become a Jedi. So it made sense to start gathering information about anyone in a planet and check who could be a candidate to be trained as Jedi.

For that reason, the Jedi Council decided to create an application to store all the people information and to be able to apply the algorithm on them. More or less, something like this:

Force Side Application

Basically, the application offered a web interface to register any person under study and storing the data in a database. In that version, the prediction algorithm was embedded as part of the management application.

For a while, everything went well, really well actually, so the Jedi Council decided to go further and to acquire a big number of TR2 T3, a droid capable to gather that same information and send it directly to the system to be analysed. With this, the Jedi Council could gather information from entire planets, so the amount of data grew in huge way.

After a while, the system started to work really bad loosing too many messages from the droids, responding very slow and even failing a lot in the predictions.

Klanus Klain realised that the problem was that the application wasn’t prepared to work with such a amount of data. So in order to save the situation, Klain decided to change the architecture for another capable to work at that scale.

Then, Klanus hired Mukanbbi, a reputed wookie with a high experience designing Reactive Systems following the DevOps principles:

Mukanbbi the wookie DevOps

Working together, they changed the original monolithic architecture for another one completely distributed. So the second version took a form like the one below:

Force Side distributed system

In the new architecture, the prediction algorithm was extracted from the management application. With this simple change, the web application could focus on its business, to register new data, and prevent to be affected for the calculus done by the prediction algorithm.

The separation in different components also allowed the new system to scale up only those component that were necessary.

So, the monolithic previous version had evolved to a system composed by two main building blocks, the transactional system, the web application basically, and the analytical system, a Machine Learning application exposed by a REST service.

With that new design, any time that the web application, the transactional system, needed to request for a prediction, the only thing that it had to do was to send a JSON message like this:

{

"name": "Manuel Lara",

"midichlorian": 13000,

"species": "human",

"gender": "male",

"homeworld": "Naboo"

}

To Which the analytical system would reply with a message quite similar to this other one:

{

"side": "Sith",

"error": "less than 17%"

}

The component responsible to handle that kind of communication was a REST Service that used a Machine Learning model generated for a separate component, the model generator, being more specific.

Thus the new analytical system was composed by two components:

A model generator: a Scala application that used the Apache Spark framework to be continuously learning a model.

application that used the framework to be continuously learning a model. A prediction service: a Java application that used Spring Boot to expose a REST service capable to make predictions using the last model generated by the previous component.

The first component was implemented using Scala just because Apache Spark has been done with that programming language.

On the other hand, the REST service was implemented with Java to take advantage of Spring Boot, one of the best microservices frameworks, and to be honest, to demonstrate that the new modular architecture was really flexible in many different ways.

It’s also interesting to mention that both components used the MLeap library, that allowed the REST service component to avoid the use of the Spark libraries, that are very heavy and extremely dependent of a Spark server.

MLeap is a great library capable to work with models created with Spark, Scikit-learn or Tensorflow. For more information follow this link.

But a Reactive System isn’t only composed by components that talk one to each other via messages, they must also be elastic and resilient. So in order to this, both components, the model generator and the prediction service were deployed as Docker containers on top of Kubernetes.

Thanks to Kubernetes, the REST service could be deployed in high availability and with an autoscaling rule that allowed to spin up a new instance if necessary. And of course, scale down the component if it was idle.

Mukanbbi, the DevOps wookie, added to the codebase all the Kubernetes resources to deploy a highly resilient and elastic prediction service, but for the other component, the model generator, the team decided to follow a different approach.

Since version 2.3.0, Spark includes a native integration with Kubernetes. So the team decided to go for that way. As a result, with this approach the model generator was launched as a executor managed by a Spark driver that only existed on Kubernetes the time taken in the model generation. After the model was published the Pods used in the generation were marked to be deleted.

For more information about the native integration between Spark and Kubernetes follow this link.

The following picture shows how was the architecture of the system inside of Kubernetes at model generation time:

Kubernetes architecture

The dotted area, the model generator, can be considered as ephemeral. It only exists while a new model is being generated.

As you could see, the new architecture was compliance with the Reactive System manifest addressing its principles:

It’s responsive because each component work without expecting a successful response from the others. For example, if the model generator couldn’t create the model bundle, the prediction service would reply with a proper error message that allows the Transactional System to work without a prediction.

because each component work without expecting a successful response from the others. For example, if the model generator couldn’t create the model bundle, the prediction service would reply with a proper error message that allows the Transactional System to work without a prediction. It’s resilient and elastic because all the component have been prepared to work in high availability and scale up and down as necessary thanks to Kubernetes . This point will be treated in much more detail in next articles.

and because all the component have been prepared to work in high availability and scale up and down as necessary thanks to . This point will be treated in much more detail in next articles. It’s message driven because all the components of the system interact one to each other via messages.

Therefore, with this new version the team was capable to build a robust system prepared to grow as necessary.