Track Real-Time Gold Prices using Apache Kafka, Pandas & MatPlotLib
By: Manoj Kukreja
Tags: Kafka, AWS, Data Science, Artificial Intelligence, Machine Learning

We are in the era where tracking, processing and analyzing real-time data is becoming a necessity of many businesses. Needless to say handling streaming data sets is becoming one of the most crucial and sought of skills for Data Engineers and Scientists.

For this article I am assuming that you are familiar with Apache Kafka — an open-source distributed event streaming platform. Apache Kafka has built-in partitioning, replication, and fault tolerance mechanisms. I have used Apache Kafka on several projects in the past for several use cases including metrics collection, log collection and stream processing.

Apache Kafka is a hot discussion topic with my students in my Data Engineering and Data Science course. Out of the many examples that I cover during the course I decided to share this one.

We will ingest streaming data from the following API into Kafka.

https://forex-data-feed.swissquote.com/public-quotes/bboquotes/instrument/XAU/USD

We will start by creating a Kafka Producer that will read data from the API above and continuously push data to Kafka.

We will also create Kafka Consumer that will continuously read data from Kafka. once the message is read we will extract the latest gold price and and plot the results on a bar chart.

Tech Stack: Apache Kafka, Pandas, Python, kafka-python library and MatPlotLib

Preparation Work

Install Java

sudo yum -y install java-1.8.0-openjdk.x86_64

Download Apache Kafka



$ sudo wget

$ sudo tar -zxf kafka_2.13-2.5.0.tgz

$ cd kafka_2.13-2.5.0 $ cd /opt$ sudo wget http://apache.forsale.plus/kafka/2.5.0/kafka_2.13-2.5.0.tgz $ sudo tar -zxf kafka_2.13-2.5.0.tgz$ cd kafka_2.13-2.5.0

Start Zookeeper and Kafka Server in background

$ cd /opt/kafka_2.13-2.5.0;export JAVA_HOME=/usr/lib/jvm/jre-1.8.0;export JRE_HOME=/usr/lib/jvm/jre $ nohup sudo /bin/zookeeper-server-start.sh config/zookeeper.properties & $ nohup sudo bin/kafka-server-start.sh config/server.properties &

Create a new Kafka Topic — Goldrates

$ sudo bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic Goldrates $ sudo bin/kafka-topics.sh --list --zookeeper localhost:2181

Clone my Github repo:

$ git clone https://github.com/mkukreja1/blogs.git

Create a Kafka Producer

Install the Kafka Producer notebook in Jupyter. I am going to run through the steps for understanding the flow.

kafka/goldrates/kafkaProducerGoldrates.ipynb

import sys

!{sys.executable} -m pip install kafka-python import time

import json

from json import dumps

from kafka import KafkaProducer

from time import sleep

import requests as req

KafkaProducer — High Level class for creating asynchronous message producer.

Declare some constants that will be used in the program. The brokers can be a list of brokers participating in your Apache Kafka cluster.

producer = KafkaProducer(bootstrap_servers=[brokers],value_serializer=lambda x: dumps(x).encode('utf-8'))

This initializes a new Kafka Producer. This producer will serialize data before sending it to the list of brokers. Data will be converted to JSON and encoded using utf-8.

while(True):

print("Getting new data...")

resp = req.get(ticker_url)

json_data = json.loads(resp.text)

producer.send(topic, json_data)

time.sleep(sleep_time)

This code runs in an never ending loop. It fetches data from the Goldrates API, coverts incoming JSON object to a Python object and finally send this data to a particular topic in Kafka. After every iteration it sleep for a period=sleep_time constant declared previously.

If everything is running fine in you Kafka Producer you should see an output as follows:

Getting new data…

Getting new data…

Leave this notebook running so that the Kafka consumer can consume these messages.

Create a Kafka Consumer

Install the Kafka Consumer notebook in Jupyter. As done previously, I am going to run through the steps for understanding the flow.

kafka/goldrates/kafkaConsumerGoldrates.ipynb

import time

import json

from kafka import KafkaConsumer

from pandas import DataFrame

from datetime import datetime

import matplotlib.pyplot as plt

KafkaConsumer — High Level class for creating asynchronous message consumer.

brokers='localhost:9092'

topic='Goldrates'

sleep_time=300

offset='latest'

Declare some constants that will be used in the program. The brokers can be a list of brokers participating in your Apache Kafka cluster. Offset=latest means the code should fetch the latest message from the Kafka queue. You also have the option of starting with the earliest.

consumer = KafkaConsumer(bootstrap_servers=brokers, auto_offset_reset=offset,consumer_timeout_ms=1000)

consumer.subscribe([topic])

This initializes a new Kafka Consumer.

auto_offset_reset — is used only if the consumer runs into unexpected problems. In this case we are asking the consumer to restart reading messages from the latest message. You also have the option of starting with the earliest.

auto_commit_interval_ms — time period between two commits.

Finally, we are asking the consumer to subscribe i.e. get messages from the topic as defined as the constant.

oldprice_dict = {}

while(True):

for message in consumer:

#print(message)

d=json.loads(message.value)

df=DataFrame(d)

for x in range(1):

#print(x)

new_price=df['spreadProfilePrices'][x][0]

ts=df['ts'][x]

#print(new_price)

print("Latest Gold Price: " + str(new_price['ask']))

datetime_time = datetime.fromtimestamp(ts/1000)

goldprice_dict[str(datetime_time)]=new_price['ask']

print("Gold Price at: " + str(datetime_time))

print("-----------------------------------------")

#print(goldprice_dict)

plt.figure(figsize=(20,10))

plt.bar(range(len(goldprice_dict)), list(goldprice_dict.values()), align='center',linewidth=0.5)

plt.xticks(range(len(goldprice_dict)), list(goldprice_dict.keys()))

plt.show()



time.sleep(sleep_time)

For every message the JSON is loaded in a Pandas Dataframe. The latest price of Gold and the timestamp are extracted from the Dataframe and stored in a Python Dictionary. Finally, the Python Dictionary is used to plot a bar chart as follows:

X Axis — Timestamp of the Gold Price Reading

Y Axis — Gold Price Value

If everything workes well, for every iteration the latest Gold Price will be plotted as follows:

Expected Output below:

Latest Gold Price: 1902.951

Gold Price at: 2020–10–26 15:16:16

— — — — — — — — — — — — — — — — — — — — -

Latest Gold Price: 1902.968

Gold Price at: 2020–10–26 15:21:17

— — — — — — — — — — — — — — — — — — — — -

Latest Gold Price: 1902.208

Gold Price at: 2020–10–26 15:26:18

— — — — — — — — — — — — — — — — — — — — -

Latest Gold Price: 1902.293

Gold Price at: 2020–10–26 15:31:19

— — — — — — — — — — — — — — — — — — — — -

Latest Gold Price: 1902.693

Gold Price at: 2020–10–26 15:36:19

— — — — — — — — — — — — — — — — — — — — -

All the code for this article can be found on the link below:

I hope this article was helpful. The topic of Apache Kafka is covered in greater detail as part of the Big Data Hadoop, Spark & Kafka course offered by Datafence Cloud Academy.