Kafka-Python explained in 10 lines of code
By: Steven Van Dorpe
Tags: Python, Kafka Python, Consumer, Producer, Kafka

Although it’s not the newest library Python has to offer, it’s hard to find a comprehensive tutorial on how to use Apache Kafka with Python. By means of approximately ten lines of code, I will explain the foundations of Kafka and it’s interaction with Kafka-Python.

Setting up the environment

First of all you want to have installed Kafka and Zookeeper on your machine. For Windows there is an excellent guide by Shahrukh Aslam, and they definitely exist for other OS’s as well.

Next install Kafka-Python. You can do this using pip or conda, if you’re using an Anaconda distribution.

pip install kafka-python conda install -c conda-forge kafka-python

Don’t forget to start your Zookeeper server and Kafka broker before executing the example code below. In this example we assume that Zookeeper is running default on localhost:2181 and Kafka on localhost:9092.

We are also using a topic called numtest in this example, you can create a new topic by opening a new command prompt, navigating to …/kafka/bin/windows and execute:

kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic numtest

What is Kafka?

Simply put, Kafka is a distributed publish-subscribe messaging system that maintains feeds of messages in partitioned and replicated topics. In the simplest way there are three players in the Kafka ecosystem: producers, topics (run by brokers) and consumers.

Producers produce messages to a topic of their choice. It is possible to attach a key to each message, in which case the producer guarantees that all messages with the same key will arrive to the same partition.

Topics are logs that receive data from the producers and store them across their partitions. Producers always write new messages at the end of the log. In our example we can make abstraction of the partitions, since we’re working locally.

Consumers read the messages of a set of partitions of a topic of their choice at their own pace. If the consumer is part of a consumer group, i.e. a group of consumers subscribed to the same topic, they can commit their offset. This can be important if you want to consume a topic in parallel with different consumers.

The offset is the position in the log where the consumer last consumed or read a message. The consumer can then commit this offset to make the reading ‘official’. Offset committing can be done automatically in the background or explicitly. In our example we will commit automatically in the background.

Let’s code

In our example we’ll create a producer that emits numbers from 1 to 1000 and send them to our Kafka broker. Then a consumer will read the data from the broker and store them in a MongoDb collection.

The advantage of using Kafka is that, if our consumer breaks down, the new or fixed consumer will pick up reading where the previous one stopped. This is a great way to make sure all the data is fed into the database without duplicates or missing data.

Create a new Python script named producer.py and start with importing json, time.sleep and KafkaProducer from our brand new Kafka-Python library.

from time import sleep

from json import dumps

from kafka import KafkaProducer

Then initialize a new Kafka producer. Note the following arguments:

bootstrap_servers=[‘localhost:9092’]: sets the host and port the producer should contact to bootstrap initial cluster metadata. It is not necessary to set this here, since the default is localhost:9092.

value_serializer=lambda x: dumps(x).encode(‘utf-8’): function of how the data should be serialized before sending to the broker. Here, we convert the data to a json file and encode it to utf-8.

producer = KafkaProducer(bootstrap_servers=['localhost:9092'],

value_serializer=lambda x:

dumps(x).encode('utf-8'))

Now, we want to generate numbers from one till 1000. This can be done with a for-loop where we feed each number as the value into a dictionary with one key: number. This is not the topic key, but just a key of our data. Within the same loop we will also send our data to a broker.

This can be done by calling the send method on the producer and specifying the topic and the data. Note that our value serializer will automatically convert and encode the data. To conclude our iteration,we take a 5 second break. If you want to make sure the message is received by the broker, it’s advised to include a callback.

for e in range(1000):

data = {'number' : e}

producer.send('numtest', value=data)

sleep(5)

If you want to test the code, it’s advised to create a new topic and send the data to this new topic. This way, you’ll avoid duplicates and possible confusion in the numtest topic when we’re later testing the producer and consumer together.

Consuming the data

Before we start coding our consumer, create a new file consumer.py and import json.loads, the KafkaConsumer class and MongoClient from pymongo. I won’t dig any deeper in the PyMongo code, since that’s outside the scope of this article.

Furthermore, you can replace the mongo code with any other code. This can be code to feed the data into another database, code to process the data or anything else you can think of. For more information about PyMongo and MongoDb, please consult the documentation.

from kafka import KafkaConsumer

from pymongo import MongoClient

from json import loads

Let’s create our KafkaConsumer and take a closer look at the arguments.

The first argument is the topic, numtest in our case.

bootstrap_servers=[‘localhost:9092’]: same as our producer

auto_offset_reset=’earliest’: one of the most important arguments. It handles where the consumer restarts reading after breaking down or being turned off and can be set either to earliest or latest. When set to latest, the consumer starts reading at the end of the log. When set to earliest, the consumer starts reading at the latest committed offset. And that’s exactly what we want here.

enable_auto_commit=True: makes sure the consumer commits its read offset every interval.

auto_commit_interval_ms=1000ms: sets the interval between two commits. Since messages are coming in every five second, committing every second seems fair.

group_id=’counters’: this is the consumer group to which the consumer belongs. Remember from the introduction that a consumer needs to be part of a consumer group to make the auto commit work.

The value deserializer deserializes the data into a common json format, the inverse of what our value serializer was doing.

consumer = KafkaConsumer(

'numtest',

bootstrap_servers=['localhost:9092'],

auto_offset_reset='earliest',

enable_auto_commit=True,

group_id='my-group',

value_deserializer=lambda x: loads(x.decode('utf-8')))

The code below connects to the numtest collection (a collection is similar to a table in a relational database) of our MongoDb database.

client = MongoClient('localhost:27017')

collection = client.numtest.numtest

We can extract the data from our consumer by looping through it (the consumer is an iterable). The consumer will keep listening until the broker doesn’t respond anymore. A value of a message can be accessed with the value attribute. Here, we overwrite the message with the message value.

The next line inserts the data into our database collection. The last line prints a confirmation that the message was added to our collection. Note that it is possible to add callbacks to all the actions in this loop.

for message in consumer:

message = message.value

collection.insert_one(message)

print('{} added to {}'.format(message, collection))

Testing

Let’s test our two scripts. Open a command prompt and go to the directory where you saved producer.py and consumer.py. Execute producer.py and open a new command prompt. Launch consumer.py and look how it reads all the messages, including the new ones.

Now interrupt the consumer, remember at which number it was (or check it in the database) and restart the consumer. Notice that the consumer picks up all the missed messages and then continues listening for new ones.

Note that if you turn off the consumer within 1 second after reading the message, the message will be retrieved again upon restart. Why? Because our auto_commit_interval is set to 1 second, remember that if the offset is not committed, the consumer will read the message again (if auto_offset_reset is set to earliest).

— Please feel free to bring any inconsistencies or mistakes to my attention in the comments or by leaving a private note. —

Acknowledgements

This article is by no means a complete guide to Kafka or Kafka-Python, but rather a comprehensive teaser that will familiarize you with essential Kafka concepts and how to transform these in useful Python code.

For more advanced topics reading the documentation is advised. If you want to deploy code, it is probably a good idea to take a look at Confluent-Kafka and this post by Russell Jurney.

Sources

Kafka-Python documentation

Consume JSON Messages From Kafka using Kafka-Python’s Deserializer

Apache Kafka documentation

Cloudera Kafka documentation

Putting Apache Kafka To Use: A Practical Guide to Building a Streaming Platform

Introducing the Kafka Consumer: Getting Started with the New Apache Kafka 0.9 Consumer Client