Realtime prediction using Spark Structured Streaming, XGBoost and Scala | by Bogdan Cojocar | Towards Data Science

In this article we will discuss about building a complete machine learning pipeline. The first part will be focused on training a binary classifier in a standard batch mode and in the second part we will do some realtime prediction.
We will use data from the Titanic: Machine learning from disaster one of the many Kaggle competitions.
Before getting started please know that you should be familiar with Scala, Apache Spark and Xgboost.
All the source code will also be available on Github. Cool, now let’s get started!
We will train a XGBoost classifier using a ML pipeline in Spark. The classifier will be saved as an output and will be used in a Spark Structured Streaming realtime app to predict new test data.
Step 1: starting the spark session
We are creating a spark app that will run locally and will use as many threads as there are cores using local[*] :
val spark  = SparkSession.builder()  .appName("Spark XGBOOST Titanic Training")  .master("local[*]")  .getOrCreate()
Step 2: defining a schema
Next we define a schema of the data we read from the csv. This is usually a better practice than letting spark to infer the schema because it consumes less resources and we have total control over the fields.
val schema = StructType(  Array(StructField("PassengerId", DoubleType),    StructField("Survival", DoubleType),    StructField("Pclass", DoubleType),    StructField("Name", StringType),    StructField("Sex", StringType),    StructField("Age", DoubleType),    StructField("SibSp", DoubleType),    StructField("Parch", DoubleType),    StructField("Ticket", StringType),    StructField("Fare", DoubleType),    StructField("Cabin", StringType),    StructField("Embarked", StringType)  ))
Step 3: reading the data
We read the csv into a DataFrame, making sure we mention we have a header.
val df_raw = spark  .read  .option("header", "true")  .schema(schema)  .csv(filePath)
Step 4: removing null values
All the null values are replaced with 0. This is not ideal, but for the purpose of this tutorial it’s ok.
val df = df_raw.na.fill(0)
Step 5: convert the nominal values to numeric
Before walking through the code on this step let’s go briefly through some Spark ML concepts. They introduce the concept of ML pipelines, which is a set of high level APIs build on top of the DataFrameswhich make it easier to combine multiple algorithms into a single process. The main elements of a pipeline are the Transformer and the Estimator. The first can represent an algorithm that can transform a DataFrame into another DataFrame, and the latter is an algorithm that can fit on a DataFrame to produce a Transformer .
In order to convert the nominal values into numeric ones we need to define aTransformer for each column:
val sexIndexer = new StringIndexer()  .setInputCol("Sex")  .setOutputCol("SexIndex")  .setHandleInvalid("keep")val cabinIndexer = new StringIndexer()  .setInputCol("Cabin")  .setOutputCol("CabinIndex")  .setHandleInvalid("keep")val embarkedIndexer = new StringIndexer()  .setInputCol("Embarked")  .setOutputCol("EmbarkedIndex")  .setHandleInvalid("keep")
We are using the StringIndexer to transform the values. For each Transformer we are defining the input column and the output column that will contain the modified value.
Step 6: assemble the columns into a feature vector
We will use another Transformer to assemble the columns used in the classification by the XGBoost Estimatorinto a vector:
val vectorAssembler = new VectorAssembler()  .setInputCols(Array("Pclass", "SexIndex", "Age", "SibSp", "Parch", "Fare", "CabinIndex", "EmbarkedIndex"))  .setOutputCol("features")
Step 7: add the XGBoost estimator
Defining the Estimator that will produce the model. The settings of the estimator can be defined in a map. We can also set up the features and the label columns:
val xgbEstimator = new XGBoostEstimator(Map[String, Any]("num_rounds" -> 100))  .setFeaturesCol("features")  .setLabelCol("Survival")
Step 8: building the pipeline and the classifier
After we created all the individual steps we can define the actual pipeline and the order of the operations:
val pipeline = new Pipeline().setStages(Array(sexIndexer, cabinIndexer, embarkedIndexer, vectorAssembler, xgbEstimator))
The input DataFrame will be transformed multiple times and in the end will produce the model trained with our data. We will save the output in order to use it in the second realtime app.
val cvModel = pipeline.fit(df)cvModel.write.overwrite.save(modelPath)
We will use Spark Structured Streaming to basically stream the data from a file. In a real world application we read the data from dedicated distributed queues such as Apache Kafka or AWS Kinesis, but for this demo we will just use a simple file.
Briefly described Spark Structured Streaming is a stream processing engine build on top of Spark SQL. It uses the same concept of DataFrames and the data is stored in an unbounded table that grows with new rows as data is streamed in.
Step 1: create the input read stream
Once again we create a spark session and define a schema for the data. Please notice that the test csv does not contain the label Survival . Finally we can create the input streaming DataFrame, df. The input path has to be a directory where we store the csv file. It can contain one or more files that have the same schema.
val spark: SparkSession = SparkSession.builder()  .appName("Spark Structured Streaming XGBOOST")  .master("local[*]")  .getOrCreate()val schema = StructType(  Array(StructField("PassengerId", DoubleType),    StructField("Pclass", DoubleType),    StructField("Name", StringType),    StructField("Sex", StringType),    StructField("Age", DoubleType),    StructField("SibSp", DoubleType),    StructField("Parch", DoubleType),    StructField("Ticket", StringType),    StructField("Fare", DoubleType),    StructField("Cabin", StringType),    StructField("Embarked", StringType)  ))  val df = spark    .readStream    .option("header", "true")    .schema(schema)    .csv(fileDir)
Step 2: load the XGBoost model
In the object XGBoostModel we load the pre trained model that will be applied for each new batch of rows we read in the stream.
object XGBoostModel {  private val modelPath = "your_path"  private val model = PipelineModel.read.load(modelPath)  def transform(df: DataFrame) = {    // replace nan values with 0    val df_clean = df.na.fill(0)    // run the model on new data    val result = model.transform(df_clean)    // display the results    result.show()  }}
Step 3: define custom ML sink
In order to be able to apply our classifier to new data we need to create a new sink (the interface between the stream and the output, which in our case is the XGBoost model). For this we need a custom sink ( MLSink ), an abstract sink provider ( MLSinkProvider ) and an implementation of the provider ( XGBoostMLSinkProvider ).
abstract class MLSinkProvider extends StreamSinkProvider {  def process(df: DataFrame): Unit  def createSink(                  sqlContext: SQLContext,                  parameters: Map[String, String],                  partitionColumns: Seq[String],                  outputMode: OutputMode): MLSink = {    new MLSink(process)  }}case class MLSink(process: DataFrame => Unit) extends Sink {  override def addBatch(batchId: Long, data: DataFrame): Unit = {    process(data)  }}class XGBoostMLSinkProvider extends MLSinkProvider {  override def process(df: DataFrame) {    XGBoostModel.transform(df)  }}
Step 4: write the data in our custom sink
The last step is to define a query that writes the data in our custom sink. A checkpoint location also has to be defined so that the application “remembers” the latest rows read in the stream in case of a failure. If we run the program each new batch of data will be displayed on the console containing also the predicted labels.
df.writeStream  .format("titanic.XGBoostMLSinkProvider")  .queryName("XGBoostQuery")  .option("checkpointLocation", checkpoint_location)  .start()