Spark Structured Streaming w/Kafka and Azure Storage
By: William Accetta
Tags: Cloud, Kafka, Spark Streaming, Apache Spark, Azure

Clients are increasing working re-architect data pipelines and ETL/ELT activities away from scheduled batch load types. Instead, the focus is on event driven, near real time architectures that allow companies to take advantage of data moments after it is available. This is crucial in limiting the latency between the arrival of data and availability to downstream consumers (business users, application, data scientist, etc).

Apache Kafka (https://kafka.apache.org/) is an open source distributed streaming platform used to build near “real time” data pipelines. Users can write and consume messages with a consistent, fault-tolerant, and proven framework. For the purposes of this demo, we are going to focus on utilizing the producer/consumer architecture with a local instance of Kafka. Then, we can consume messages by pairing Kafka with Spark Structured Streaming (https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html). Developer can use Spark Streaming, but with Spark 2.x, Apache Spark unveiled a DataFrame driven process that proves to be user-friendly. Using spark structured streaming, developers can interact with Kafka streams with DataFrame or DataSet operations. Simply, the spark job will “listen” the prescribed topic and process data accordingly.

The last portion of this example is a cloud based storage destination. In this case, the landing zone is a Microsoft Azure Blob Storage Container however, the concepts displayed here remain true for AWS S3, Google Cloud Storage, and IBM Cloud Object Storage. Simply, all object storage options manage data as “objects” rather than a traditional file system. A pro of this type of storage is high availability, low cost, and ability to distinguish between storage tier access levels. Microsoft does allow users to interact is use object storage with Azure Data Lake Storage Gen2, which adds a hierarchal namespace on-top of blob containers. This solution combines capabilities of both Blob Storage and Azure Data Lake Storage Gen1.

Demo: In this example, we will create a Kafka producer and topic using a python script and an openly available API. Next, using a spark-scala script the topic value will be written to Blob Storage. Note that this demo will require an Azure Account, but as the demo only utilizes raw storage it will be minimal to no costs.

Demo Requirements:

This project was constructed off my local computer (MacOS Catalina) with a few additional software requirements. This demo will not take users though installation. If users need assistance please the open source documentation for installation.

Spark: 2.4.4

Kafka: 2.12

Python: 3.7

Scala: 2.12

Microsoft Visual Studio Code

Intellj IDEA

Microsoft Azure Account

Setting up the Kafka producer — python:

The script utilizes the openweathermap API (https://openweathermap.org/api) to connect to a public available stream. You can create a free account and utilize the current weather and forecast collection which provides free access to an API. Then, using the cityID function, users can select specific cities as desired. Before executing the script, the Kafka topic needs to be created.

./kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic weatherdata

Then the python script can be executed using Visual Studio Code.

# -*- coding: utf-8 -*- """ Created to generate a Kafka steaming process """ import os import json import requests import schedule import time from kafka import KafkaProducer from json import dumps def job(): output = '' l =['5391959','4351977','5128581','4699066','5368361','4896861','4560349','5308655','4684888','5809844','4164138','5419384','4930956','4180439','4053671'] for i in l: url3 = 'http://api.openweathermap.org/data/2.5/group?id='+i+'&appid={insert your appID}' json_data = requests.get(url3).json() producer = KafkaProducer(bootstrap_servers=['localhost:9092'], value_serializer=lambda x: dumps(x).encode('utf-8')) producer.send("weatherdata", (json_data)) # Must create topic before this step schedule.every(1).minutes.do(job) while 1: schedule.run_pending() time.sleep(1)

This python script will generate a streaming process in a topic labeled “weatherdata” running on the local computer. As a user, this can be verified in the terminal using the following command.

./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic weatherdata

Building a Storage Container:

Before attacking the spark portion, let’s create the storage container. From the Microsoft Azure Documentation (https://azure.microsoft.com/en-us/pricing/details/storage/blobs/):

Total cost of block blob storage depends on:

Volume of data stored per month.

Quantity and types of operations performed, along with any data transfer costs.

Data redundancy option selected.

In follow on posts, I will take you through the process of building cloud infrastructure with the HashiCorp product Terraform. However, for the case of this demo I would recommend creating a storage account with a blob container in the portal (https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-portal). In the image below, my storage container is illustrated. I would recommend using Microsoft Storage explorer for interacting with the data layer (https://azure.microsoft.com/en-us/features/storage-explorer/). I would recommend signing in with your Microsoft account credentials.

Spark-Structured Streaming:

Finally, utilizing Spark we can consume the stream and write to a destination location. Yes, in most business applications there is not a need to both create and consume the stream. However, given how this shows the end to end process, we will perform both. Now, spark supports both python and scala variations. However, spark scala proves to both perform better and include new features sooner than pyspark based code. This article will not be a study between scala and pyspark but will acknowledge that either method could be suitable.

Using IntellJ and a SBT project, we can construct a script that reads the spark topic and writes it to Blob Storage.

Then, since the goal of this tutorial is to link spark structured steaming with Azure Blob Storage, we are going to take this step by step. The first step is to establish the maven packages required.

name := "sparkStreamingAzure"



version := "0.1"



scalaVersion := "2.12.0"



// https://mvnrepository.com/artifact/org.apache.spark/spark-core

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.4.4"



// https://mvnrepository.com/artifact/org.apache.spark/spark-sql

libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.4"



// https://mvnrepository.com/artifact/org.apache.spark/spark-sql-kafka-0-10

libraryDependencies += "org.apache.spark" %% "spark-sql-kafka-0-10" % "2.4.4"



// https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-azure

libraryDependencies += "org.apache.hadoop" % "hadoop-azure" % "2.7.7"



// https://mvnrepository.com/artifact/com.microsoft.azure/azure-storage

libraryDependencies += "com.microsoft.azure" % "azure-storage" % "2.0.0"

Next, we need to build our spark session. You will see we are using the local compute option.

val spark = SparkSession

.builder()

.master("local[*]")

.appName("sparkStreamingAzure")

.getOrCreate() println("Started Spark Session")

Then, we need to configure our connection to our Azure Environment. You will need your access key associate with the storage account.

val yourKey = "{insert your Storage Key}"



spark.sparkContext.hadoopConfiguration.set("fs.azure", "org.apache.hadoop.fs.azure.NativeAzureFileSystem") spark.sparkContext.hadoopConfiguration.set("fs.azure.account.key.accettadev.blob.core.windows.net", yourKey)

The next step is setting up the Kafka configurations. There are a few items to call our during this step. First, notice that we use the same topic name and Kafka brokers as utilized earlier. Then, we set a destination which is our Azure Storage account. Next, we set a checkpoint location on our local computer. The checkpoint can also be set on the cloud storage account. These features allow streams to recover from failure and saves the state of the process when last stopped. Lastly, the select statement extracts only the data value from the message and converts the value from a binary to string. The columns available to operate with and their respective data types is shown below. This information is gathered from the spark-structured-streaming-kafka-integration guide (https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html). Our raw data is json and we are storing it in the column that shares the same name.

// The Kafka broker hosts and topic used to write to Kafka

val kafkaBrokers="localhost:9092"

val kafkaTopic="weatherdata" // Stream from Kafka

val kafkaStreamDF = spark.readStream.format("kafka").option("kafka.bootstrap.servers", kafkaBrokers).option("subscribe", "weatherdata").option("failOnDataLoss", false).option("startingOffsets", "latest").load()



val format = "json"

val destination = "wasbs://container>@<storage_account.blob.core.windows.net/<container>/weatherdata"

val streamcheckpoint = "$HOME/Spark-streaming/weatherCheckpoint"

val df = kafkaStreamDF.selectExpr("CAST(value AS STRING) as json")

Our last step is writing the stream to the destination. There are obviously more functionality and capabilities of Spark-structured streaming which is not utilized in this example. The goal is simply to get data to our storage blob. It is worth to note that spark streaming supports a variety of structured and semi-structured data types. Best practices would suggest that users select a Hadoop data type like Avro or parquet. Avro is a row based format that supports schema evolution. Then, parquet is columnar storage option which is favorable when compressing data.

val consoleOutput = df.writeStream

.outputMode("append")

.format(format) // can be "orc", "json", "csv", "console", "parquet", "text"

.option("path",destination)

.option("checkpointLocation", streamcheckpoint)

.option("failOnDataLoss", false)

.start()

consoleOutput.awaitTermination()

Now let’s tie the entire piece of code together!

import org.apache.spark.sql.SparkSession

import org.apache.spark.sql.types._



object sparkStreamingAzure {

def main(args: Array[String]): Unit = {



import org.apache.log4j._

Logger.getLogger("org").setLevel(Level.ERROR)

Logger.getLogger("akka").setLevel(Level.ERROR) val spark = SparkSession

.builder()

.master("local[*]")

.appName("sparkStreamingAzure")

.getOrCreate() println("Started Spark Session") val yourKey = "{insert your Storage Key}"



spark.sparkContext.hadoopConfiguration.set("fs.azure", "org.apache.hadoop.fs.azure.NativeAzureFileSystem") spark.sparkContext.hadoopConfiguration.set("fs.azure.account.key.accettadev.blob.core.windows.net", yourKey) // The Kafka broker hosts and topic used to write to Kafka

val kafkaBrokers="localhost:9092"

val kafkaTopic="weatherdata" // Stream from Kafka

val kafkaStreamDF = spark.readStream.format("kafka").option("kafka.bootstrap.servers", kafkaBrokers).option("subscribe", "weatherdata").option("failOnDataLoss", false).option("startingOffsets", "latest").load()



val format = "json"

val destination = "wasbs://<container>@<storage_account>.blob.core.windows.net/<container>/weatherdata"

val streamcheckpoint = "$HOME/Spark-streaming/weatherCheckpoint"

val df = kafkaStreamDF.selectExpr("CAST(value AS STRING) as json")



// Select data from the stream and write to Blob Storage

val consoleOutput = df.writeStream

.outputMode("append")

.format(format) // can be "orc", "json", "csv", "console", "parquet", "text"

.option("path",destination)

.option("checkpointLocation", streamcheckpoint)

.option("failOnDataLoss", false)

.start()

consoleOutput.awaitTermination()

}

}

This allows a user to write a stream from a Kafka topic to Microsoft Azure Blob Storage. From here, the possibilities are almost endless. In later blog I will explore using Azure Functions, linking to Snowflake Data-Warehouse-as-a-Service, and using server-less ETL.

Sources:

https://databricks.com/blog/2017/04/04/real-time-end-to-end-integration-with-apache-kafka-in-apache-sparks-structured-streaming.html