Kafka No Longer Requires ZooKeeper | by Giorgos Myrianthous | Towards Data Science

Apache Kafka 2.8.0 is finally out and you can now have early-access to KIP-500 that removes the Apache Zookeeper dependency. Instead, Kafka now relies on an internal Raft quorum that can be activated through Kafka Raft metadata mode. The new feature simplifies cluster administration and infrastructure management and marks a new era for Kafka itself.
Zookeeper-less Kafka
In this article we are going to discuss why there was a need for removing ZooKeeper dependency in the first place. Additionally, we will discuss how ZooKeeper has been replaced by KRaft mode as of version 2.8.0 and also explore the impact of removing this dependency and how Kafka itself will benefit from this enhancement.
Until now, Apache ZooKeeper was used by Kafka as a metadata store. Metadata for partitions and brokers were stored to the ZooKeeper quorum that was also responsible for Kafka Controller election.
ZooKeeper is an external system to Kafka which means that it comes with its own configuration syntax, management tools and best practices. Therefore, if you wanted to deploy a Kafka cluster then you would also had to manage, deploy, and monitor Zookeeper as well. Since these two distributed systems require different configuration the level of complexity increases and thus it is much easier for system administrators to make mistakes. Having two systems also leads to duplication of work— for example, in order to enable security features you would have to apply the relevant configuration to both services.
Having an external metadata store is inefficient in terms of resources as you need to run additional processes. Furthermore, this limits the scalability of Kafka itself. Every time the cluster is starting up, the Kafka controller must load the state of the cluster from ZooKeeper.
The same happens when a new controller is being elected. Given that the amount of metadata gets bigger over time, this means that the loading of such metadata becomes more inefficient over time leading to high loading processes and thus it limits the number of partitions the cluster can store.
KPI stands for Kafka Improvement Proposals and KPI-500 introduces the fundamental architecture of ZooKeeper-less Kafka.
Version 2.8.0 introduces an early access to Zookeeper-Less Kafka as part of KPI-500. Note though that the implementation is partially complete and thus you should not use it in production environments.
In the latest release, ZooKeeper can be replaced by an internal Raft quorum of controllers. When Kafka Raft Metadata mode is enabled, Kafka will store its metadata and configurations into a topic called @metadata. This internal topic is managed by the internal quorum and replicated across the cluster. The nodes of the cluster can now serve as brokers, controllers or both (called combined nodes).
When KRaft mode is enabled, only a few selected servers can serve as controllers and these will compose the internal quorum. The controllers can either be in active or standby mode that will eventually take over if the current active controller server fails or is taken down.
Now every Kafka Server comes with an additional configuration parameter called process.roles. This parameter can take the following values:
broker: The Kafka Server will serve as a broker
controller: The Kafka Server will serve as a controller of the internal Raft quorum
broker,controller: The Kafka Server will serve as both a controller of the quorum and a broker
Note that when process.roles is not provided at all, it is assume that the cluster will run in ZooKeeper mode. Therefore, for time being process.roles configuration parameter is the only way you can activate KRaft mode.
Additionally, each node is now identified with its node.id and must now provide controller.quorum.voters configuration parameter which is the equivalent to zookeeper.connect in ZooKeeper mode. This parameter is used to identify the controller servers of the internal quorum and the format used for the values is serverID@host:port, serverID@host:port, ....
Now let’s assume that within our Kafka Cluster which is running in KRaft mode we have 7 brokers and 3 controllers. The snippet below demonstrates an example configuration for one of the controller servers in the Raft quorum.
process.roles=controllernode.id=1listeners=CONTROLLER://controller-1-host:9092controller.quorum.voters=1@controller-1-host:9092,2@controller-2-host:9092,3@controller-3-host:9092
Likewise, the below configuration demonstrates how to setup one of the brokers of the cluster:
process.roles=brokernode.id=4listeners=PLAINTEXT://:9092controller.quorum.voters=1@controller-1-host:9092,2@controller-2-host:9092,3@controller-3-host:9092
The removal of Apache ZooKeeper dependency is definitely a step forward for the platform. The whole community (and I’d say Confluent in particular) has been working towards this direction for the last few years. The earliest release has been a tremendous effort of the whole Kafka community that is still working on improvements so that ZooKeeper-Less Kafka mode is feature-complete within this year.
We’ve been headed this direction for years
— Jason Gustafson @ Kafka Summit San Francisco 2019
The removal of Apache ZooKeeper dependency simplifies the infrastructure management for Kafka deployments. Kafka and ZooKeeper are two distinct services — now that Kafka has been unified so that it is not dependent on external services to server as metadata stores, the learning curve will be shortened and this will eventually help broaden Kafka’s adoption.
Furthermore, this enhancement offers a more scalable and robust overall architecture. As discussed earlier, in ZooKeeper mode Kafka had to store its metadata into ZooKeeper nodes. Every time the cluster was starting up or a controller election was happening, Kafka Controllers had to read the metadata from an external service which was inefficient. By replacing ZooKeeper with this internal Raft quorum, deployments can now support more partitions.
Removing the ZooKeeper dependency also enables the support of clusters with single node. When you want test Kafka as part of a proof of concept you no longer have to start multiple processes.
The removal of Zookeeper dependency is a huge step forward for Kafka. In fact, the new KRaft mode feature will extend scalability capabilities of Apache Kafka and also shorten the learning curve since now teams won’t have to worry about ZooKeeper any longer. It will also make Kafka configuration and deployment way easier and more efficient.
In addition to KPI-500, Kafka 2.8.0 comes with numerous improvements and bug fixes so make sure to take a look at the release notes.
Finally, I want to highlight once again that this is currently an early-access which means it should not be used in production environments.