Streaming ML CI/CD in a few days: how we improved our easy-to-use code delivery system for Spark Streaming applications and ML
By: Eugene Lopatkin
Tags: Machine Learning, Spark, Spark Streaming, Apache Spark, Kubernetes

Streaming ML CI/CD in a few days: how we improved our easy-to-use code delivery system for Spark Streaming applications and ML Eugene Lopatkin Mar 22·3 min read

Overview

A significant trend in the development of IT business in now days is readiness to work with hot data, the lifetime of which from the moment of its appearance can be less than a second.

Let’s say you come to a store and take out a loan to buy a phone. You want to get a loan on favorable terms. And the bank wants to give a loan to a verified client. The time window in which you need credit money is relatively short. An example from Telecom domain. You have run out of money, and at this time your wife is going to deliver a baby. You need to call an ambulance. A trust payment from an operator would be very helpful.

Cost of data for business in such situations is extremely important. This state of affairs in the market has led to the emergence of kappa architecture and streaming data handling, which should guarantee minimal latency.

Recently I wrote about our code delivery system based on Kubernetes, Airflow and Jenkins.

Among the advantages of our system:

Extremely low development costs Solving all required code delivery tasks Single point of product management Fault tolerance and monitoring point

For streaming support we want to keep the same way of automation, the same point of control as batch jobs and minimize an amount of using resources.

Concepts

We decided that automation of streaming applications should not be different from automation of batch applications. In addition, we want to build Scala application “on the fly” from repository.

The fact is that Spark Streaming applications block the console after executing the spark-submit. The option

spark.yarn.submit.waitAppCompletion = false

allows us to achieve a spark-submit behavior identical to the batch job submission. Accordingly, after submitting such an application our Kubernetes Pod, under which submitting is happened, collapses and frees up resources.

Next, we have to decide how to understand that the job is running. It is quite simple to do through YARN CLI. This bash command does it:

while yarn application -list | grep InternationalRoamingApp | grep -e RUNNING -e ACCEPTED; do sleep 10; done && exit 1 > /dev/null 2>&1

If application fails we also should to rerun spark-submit steps to restart application.

DAG generation remastering

Now when we’ve found out how Spark jobs should be handled. We should just add SSH operator generation support for our automation system which will allows to check job state.

YAML Configuration files

New structures of Single-step and multi-step files. Rework allows declaring command for remote execution (in our case it is a Hadoop cluster) and additional containers for Pod that could mix in some additional logic (in our case application building).

Single-step configuration file

Multi-step configuration

New in j2 DAG template

Method that allows to restart all steps before failed step:

Conversion for commands for internal usage in Python:

Loops for sidecar containers:

Macros for SSH Operator that executes a remote command:

Resulting Python script from a rendering of this template you could see in Appendix 1.

DAG execution remastering

At the stage of DAG execution, the first step is a spark-submit that runs a pod, which completes an assembly and a submission. After that, the usual SSH operator starts, which controls the launched application. If the application crashes, the SSH operator command crashes, and the flow restarts. We could control number of restarts and restart interval of our streaming application.

New in j2 pod generation command template

Command allows adding sidecar containers. In our case, we are using one to build Scala projects.

Rendering result of this command is in Appendix 2.

ML Model composition in Spark Streaming applications

Spark allows you to use models as:

UDF Function Transformer or Estimator inside Spark ML pipeline.

Our system allows to split model and data preparation logic, use models as separate microservices and make compositions in many different ways.

Appendix 1. DAG generation example

Appendix 2. Rendered command example