Building Python Microservices with Apache Kafka: All Gain, No Pain | by Krasnov Vitaliy | Towards Data Science

Engineers often use Apache Kafka in their everyday work. The major tasks that Kafka performs are: read messages, process messages, write messages to a topic, or aggregate messages for a certain period of time. The window can be fixed (e.g. hourly aggregation) or sliding (e.g. hourly aggregation starting from a certain point in time).
Given the complexity of Kafka work, I do not recommend that you try to build your own solution from scratch. It will be too hard to fine-tune, test, and support. Fortunately, there are ready-to-roll implementations like Kafka Streams for Java and Kafka Streams for Python — Python Faust by Robinhood. The team also has a fork supported by the community — Faust Streaming.
This article shares my experience of building asynchronous Python microservices that “communicate” using Apache Kafka at Provectus. Faust, a stream processing library that ports the ideas from Kafka Streams to Python, is used as a microservices foundation. Faust also provides an HTTP server and a scheduler for interval and scheduled tasks. In a test implementation, I will use such instruments and libraries as FastAPI, Grafana, and Prometheus as well.
Faust is the implementation of Kafka Streams on Python. Originally developed by Robinhood, it is still used as their default library for high-performance distributed systems and real-time data pipelines.
The library works with Python 3.6+ and Kafka 0.10.1+, and supports various modules for data storage, data collection, and acceleration. Faust enables you to take advantage of such common Python libraries as NumPy, SciPy, TensorFlow, SQLAlchemy, and others.
I believe that to truly learn how to do something, you should actually do it, step by step. Below is an architectural diagram of a simple test project we are going to build.
Test data will be compiled in demo_server microservice while data_requester microservice will be requesting data every second. data_requester is the first microservice in our Faust system. Once data is requested, data_requester will push the reply to Kafka. Then, this message is read, processed, and pushed back to Kafka by data_processor. Following that, it is read by data_aggregator and db_loader, where the first microservice calculates average values and the second one loads them into a database. db_loader also loads the messages that are being generated by data_aggregator. In the meantime, api_gateway can pull the data from the database when it is requested by a user. All activities are monitored using Prometheus and can be visualized in Grafana. The project and additional services are launched in docker-compose.
As we are going to spin up the project using docker-compose, it makes sense to describe all third-party services. We will need Apache Kafka, a database, an admin dashboard to manage it, and a couple of monitoring services (Prometheus and Grafana). Let’s use Postgres as our database, and PgAdmin as its components for administration and monitoring.
At this stage, our docker-compose.yml looks like this:
docker-compose, along with the infrastructure, configurations, and additional scripts, is stored in a separate folder of the repository.
Although every service has its own tasks and is written using two different libraries — FastAPI and Faust — they have several common components.
First is the loader of configs, based on Configloader library. It enables you to load configs using yaml files and to redefine their values through variable environments. This is useful in situations where you need to redefine specific values in your docker container.
Second is Prometheus exporter. You can use it to export data and metrics, and to display them in Grafana. Bear in mind that its implementation in FastAPI and Faust differs a bit.
To test and demonstrate the system, we will use a simple service that returns the values for currency pairs in a JSON format. For this, we will also use FastAPI library. The service is as easy to use as Flask, but it comes with async and Swagger UI documentation out of the box. It is available here: http://127.0.0.1:8002/docs
We will specify just one request for the service:
As you can see from the code above, every request returns a JSON with two pairs of randomly generated values. Here is what the reply may look like:
{ “USDRUB”: 85.33, “EURRUB”: 65.03}
To monitor the values, add Prometheus to the system. It should be added as middleware with a separate specified path.
You can check out the metrics for your values in the browser: http://127.0.0.1:8002/metrics
Data from the demo server is requested by api_requester microservice with the help of Faust. Because Faust is asynchronous, we can simply use the aiohttp client to request test data.
To begin, let’s create a Faust application.
When creating the application, make sure that you specify its name — a mandatory argument. If we launch several versions of the service with the same name, Kafka will distribute partitions between all of them, enabling us to scale our system horizontally. Next, we need to specify the method of message serialization in value_serializer. In this example, we can read raw as is, to serialize received messages afterwards. We should also define the address and port to access our HTTP server hosted by Faust.
Next, we need to define the topic that is going to receive the replies from demo_server.
The first argument is the topic name; it is mandatory. Then, you can specify the number of partitions (optional). Bear in mind that the number of partitions must be similar to the number of topic partitions in Kafka (specified when the topic is created).
Note that data_requester service does not read messages from topics, but it continually pushes requests to the data emulator and then processes the replies. To enable this, you need to write a real-time clock function that is triggered every second, or at specified intervals.
We can see that the function is performed periodically by @app.timer decorator. It receives intervals per second as an argument. Then, the function generates a class instance for DataProvider that is responsible for providing replies to requests. After every consequent request, the timer’s values increase in Prometheus. If data is received per request, it is sent to a topic. Because Kafka works with specific keys and in-byte messages, we need to serialize our data before pushing it to the topic.
You will also need to initialize Prometheus when starting your application. For that, you can simply use a function that you call when starting the app.
Bear in mind that the function is assigned and defined by @app.task decorator. Prometheus starts as a separate server using its own port, which works in parallel with Faust’s HTTP server.
Here you go! Making requests and writing data to Kafka on a per-second basis is not that hard. Adding the monitoring component can be quickly done, too.
Our next microservice — data_processor — processes the pairs of data that are being received by api_requester microservice. The code for application initialization and data monitoring is identical to the code we used to manage data_requester. However, in the case of data_processor, the service receives messages from a topic to process them. It can be done with a function.
The function is based on the @app.agent(src_data_topic) decorator that tells the function to process messages in src_data_topic. The messages are read in async for msg_key, msg_value in stream.items().
Then, we need to serialize the received message, and extract currency pairs and their values. Every pair should be separately written to a consequent topic.
The microservices for requesting and processing data read and write data in Kafka as streams. Every new message is processed independently of previous messages. But, once in a while, you may need to process a certain number of messages simultaneously. Just imagine that you need to find an average for the last ten pairs of values. For doing that, your system should be able to store these ten pairs somewhere. You cannot store data locally: if you trigger several versions of the aggregation service, each of them will locally store only its values, which, as a result, will lead to incorrect calculation of averages. Here is where you can take advantage of Faust tables.
The tables store the values in changelog topic and locally — in rocksdb. This allows all versions of the service to work in sync. Upon reboot, it restores the state from a local database and, when an available changelog is read, continues to work as is.
The topic’s name for a table is assigned like this: <service-name>-<table-name>-changelog. In the case of our system, the name is: data-aggregator-average-changelog.
Processing and storing new messages in the table is quite simple:
As seen from the code above, you need to define how your function processes messages in the topic. Every new value received should be stored in the table to calculate an average. You can use the table in ways similar to a standard Python dictionary.
db_loader microservice reads two topics — data-aggregator-average-changelog and processed_data — at once. It writes messages from data_aggregator to the first topic and from data_processor to the second one. This is why we need to describe two functions for message processing.
Similar to other services, you need to: read the message, store it in a database, update the metrics. To manage the database, we will use ORM SQLAlchemy. Note that you should set it up to work asynchronously. For that, define the required dependencies.
asyncpg==0.23.0SQLAlchemy==1.4.0
Specify DB URI in the config.
DB_URI: “postgresql+asyncpg://postgres:postgres@127.0.0.1:5432/currencies”
Use asynchronous sessions in the database management code.
Finally, let’s create a simple service using FastAPI to request the results from the database. It will read the results from the database and push them to JSON. To manage the database, you can use ORM (as we did before). As to FastAPI, the instructions are similar to Step 2. Data Emulation Microservice.
Building microservices with Apache Kafka on Python is quite simple. You can offload almost all of the work for handling and managing Kafka to Faust. Then, just describe and define the functions to see them processing your messages in real time.
I hope this tutorial has helped you get a better understanding of how you can marry microservices, Apache Kafka, and Python. If you are interested in monitoring and management of clusters in Apache Kafka, I recommend that you also check out this article.
Any questions? Please, reach out to me for discussion in the comment’s section.