Introducing kafkaQ: a real-time monitoring tool for Kafka applications
By: Shreshth Srivastava
Tags: Pub Sub, Kafka, Distributed Systems, JavaScript, Nodejs

A critical gap in monitoring capabilities

In 2011, LinkedIn developed Apache Kafka — an event streaming platform that uses the publish-subscribe (pub-sub) architecture. Since then, companies like Twitter, Netflix, and Uber have integrated Kafka to scale their distributed systems. With Kafka’s design, companies are able to process high volumes of event-driven data, such as clicks, searches, etc. For more information about the Kafka architecture, see this article.

Naturally, Kafka applications generate large amounts of data. These analytics, however, can enhance more than just product decisions. In fact, if a company has an intimate familiarity with their system’s performance, operations, and weak points, they can rapidly improve the user experience.

For this reason, an effective Kafka monitoring tool can supercharge a company’s engineering process. So far, however, this field is lacking, which has created frustration for Kafka developers.

How does kafkaQ solve this problem?

kafkaQ aims to solve the need for a robust, open-source Kafka monitoring tool. Our Electron app features a dashboard that allows you to track your system’s performance in real-time. Individually, we track each of the following metrics for both producers and consumers:

Data Size — the volume of data flowing through a given producer/consumer at any point in time (in bytes)

— the volume of data flowing through a given producer/consumer at any point in time (in bytes) Processing Time — the amount of time your system uses to process each request (in milliseconds)

— the amount of time your system uses to process each request (in milliseconds) Pending Duration — the quantity of time each request spends in limbo before being processed (in milliseconds)

Under the hood, kafkaQ uses the KafkaJS API to connect with your Node.js backend and extract data. This information is then routed through an Express server to the Electron UI via WebSocket. However, all these details are abstracted away from the user through our NPM package. For this reason, kafkaQ can be integrated into a preexisting Kafka application in a matter of minutes.

Installation

Now that we understand the problem kafkaQ aims to solve, along with the functionality it provides, let’s see how to integrate kafkaQ with your application! In total, it takes just 4 lines of code to be able to view our metrics.

To get started, you’ll need to install our NPM package. To do so, run npm install kafkaq-monitor

Now, let’s track your producer analytics. In your producer file, you’ll need to import our trackProducer method (see line 2 of the code snippet below), and then invoke it (see line 12) just after connecting your producer (make sure to pass your producer as an argument). See the example below:

Similarly, in your consumer file, import our trackConsumer method (see line 2 of the code snippet below), and invoke it (see line 20) after your consumer subscribes to a topic. Again, pass your consumer in as an argument. Here’s an example:

And that’s it! You’re done integrating kafkaQ with your application. To view your metrics, simply run npm run electron-app and you should be able to track your system in real-time. Here’s an example of how your dashboard should look:

Your kafkaQ dashboard

A final note

The kafkaQ team welcomes any suggestions, so if you find any bugs or have other feedback, feel free to submit a GitHub issue here. We’ve also written thorough documentation which you can find on our companion website here. If you’ve benefited from kafkaQ or are inspired by our vision, we sincerely appreciate each GitHub star we receive.

kafkaQ Engineers:

Jonathan Barenboim, GitHub & LinkedIn

Kai-Wei Hsu, GitHub & LinkedIn

Micah Y. Turan, GitHub & LinkedIn

Shreshth Srivastava, GitHub & LinkedIn