Streaming Data from Apache Kafka Topic using Apache Spark 2.4.7 and Python
By: Sandeep Kattepogu
Tags: Microsoft Sql Server, Change Data Capture, Apache Kafka, Apache Spark, Python

Press “CTRL + C” to end the Spark context.

Step 5: Running Own Functions on Output

While printing aggregated CDC data is interesting, it is hardly useful. If you want to run your own functions (whether to store the information on the Spark node or stream it elsewhere), changes need to be made to the completed file. One way to do it is to substitute the “pprint()” function for “foreachRDD” so that each reduced set of fruit and totals can have a function run on them.

# To program your own behavior, change this...

counts = dks.map(lambda x: json.loads(x[1])).flatMap(lambda dict: dict.items()).filter(lambda items: items[0]=="payload").map(lambda tupler: (tupler[1]["after"]["fruit_name"], tupler[1]["after"]["num_sold"])).reduceByKey(lambda a, b: a+b).pprint() # To this...

counts = dks.map(lambda x: json.loads(x[1])).flatMap(lambda dict: dict.items()).filter(lambda items: items[0]=="payload").map(lambda tupler: (tupler[1]["after"]["fruit_name"], tupler[1]["after"]["num_sold"])).reduceByKey(lambda a, b: a+b).foreachRDD(somefunction)

Once this is done, custom functions can be run by replacing “somefunction” above with the function name. Here is an example function that will do the same behavior as “pprint()”, but, by virtue of the format the Kafka data is read into Spark, will leave out superfluous timestamps.

def printy(a, b):

listy = b.collect()

for l in listy:

print(l) counts = dks.map(lambda x: json.loads(x[1])).flatMap(lambda dict: dict.items()).filter(lambda items: items[0]=="payload").map(lambda tupler: (tupler[1]["after"]["fruit_name"], tupler[1]["after"]["num_sold"])).reduceByKey(lambda a, b: a+b).foreachRDD(printy)

Using a custom function to leave out timestamps

Notice that there are four different aggregation events with no timestamps between them and prints nothing if no insertions happen. With a little bit of editing this function can export these values to a separate program that can track the totals for each fruit over different spans of time. This will be covered in the final part of this tutorial.

Step 6: Changing the Spark Job to Filter out Deletes and Updates

Updates and deletes are not considered. If you require updates and deletes to be filtered out, it will take some work with Python logic and some extra filtering of the JSON data. This will be based on the “op” parameter found at the end of each JSON data string.

Operation parameter for inserting a new row

Operation parameter for updating a row

Operation parameter for deleting a row

Completed Python File

The below file, when submitted as a Spark job with /etc/spark/bin/spark-submit — packages org.apache.spark:spark-streaming-kafka-0–8_2.11:2.2.3,org.apache.spark:spark-sql-kafka-0–10_2.11:2.2.3 readkafka.py , takes in all new CDC data from the Kafka topic every two seconds. In the case of the “fruit” table, every insertion of a fruit over that two second period will be aggregated such that the total number value for each unique fruit will be counted and displayed.

#Imports and running findspark

import findspark

findspark.init('/etc/spark')

import pyspark

from pyspark import RDD

from pyspark import SparkContext

from pyspark.streaming import StreamingContext

from pyspark.streaming.kafka import KafkaUtils

import json #Spark context details

sc = SparkContext(appName="PythonSparkStreamingKafka")

ssc = StreamingContext(sc,2) #Creating Kafka direct stream

dks = KafkaUtils.createDirectStream(ssc, ["testDB.dbo.fruit"], {"metadata.broker.list":"{replace with your Kafka private address}:9092"}) # Transforming CDC JSON data to sum fruit numbers

# based on fruit name

def printy(a, b):

listy = b.collect()

for l in listy:

print(l) counts = dks.map(lambda x: json.loads(x[1])).flatMap(lambda dict: dict.items()).filter(lambda items: items[0]=="payload").map(lambda tupler: (tupler[1]["after"]["fruit_name"], tupler[1]["after"]["num_sold"])).reduceByKey(lambda a, b: a+b).foreachRDD(printy) #Starting Spark context

ssc.start()

ssc.awaitTermination()

Addendum

In the next part of this tutorial, we will install Grafana, Graphite Carbon, and Graphite Web onto an Ubuntu 18.04 EC2 instance to stream and plot the CDC data transformed by Spark. The Spark Python job from this tutorial will also be edited to use StatsD to interface with Graphite Carbon. A link will be added HERE when Part 3 is available.