Chapter 2. Using Kafka in KRaft mode KRaft (Kafka Raft metadata) mode replaces Kafka's dependency on ZooKeeper for cluster management. KRaft mode simplifies the deployment and management of Kafka clusters by bringing metadata management and coordination of clusters into Kafka. Kafka in KRaft mode is designed to offer enhanced reliability, scalability, and throughput. Metadata operations become more efficient as they are directly integrated. And by removing the need to maintain a ZooKeeper cluster, there's also a reduction in the operational and security overhead. To deploy a Kafka cluster in KRaft mode, you must use Kafka and KafkaNodePool custom resources. The Kafka resource using KRaft mode must also have the annotations strimzi.io/kraft: enabled and strimzi.io/node-pools: enabled . For more details and examples, see Section 8.2.1, "Deploying a Kafka cluster in KRaft mode" . Through node pool configuration using KafkaNodePool resources , nodes are assigned the role of broker, controller, or both: Controller nodes operate in the control plane to manage cluster metadata and the state of the cluster using a Raft-based consensus protocol. Broker nodes operate in the data plane to manage the streaming of messages, receiving and storing data in topic partitions. Dual-role nodes fulfill the responsibilities of controllers and brokers. Controllers use a metadata log, stored as a single-partition topic ( __cluster_metadata ) on every node, which records the state of the cluster. When requests are made to change the cluster configuration, an active (lead) controller manages updates to the metadata log, and follower controllers replicate these updates. The metadata log stores information on brokers, replicas, topics, and partitions, including the state of in-sync replicas and partition leadership. Kafka uses this metadata to coordinate changes and manage the cluster effectively. Broker nodes act as observers, storing the metadata log passively to stay up-to-date with the cluster's state. Each node fetches updates to the log independently. If you are using JBOD storage, you can change the volume that stores the metadata log . Note The KRaft metadata version used in the Kafka cluster must be supported by the Kafka version in use. Both versions are managed through the Kafka resource configuration. For more information, see Section 10.2, "Configuring Kafka in KRaft mode" . In the following example, a Kafka cluster comprises a quorum of controller and broker nodes for fault tolerance and high availability. Figure 2.1. Example cluster with separate broker and controller nodes In a typical production environment, use dedicated broker and controller nodes. However, you might want to use nodes in a dual-role configuration for development or testing. You can use a combination of nodes that combine roles with nodes that perform a single role. In the following example, three nodes perform a dual role and two nodes act only as brokers. Figure 2.2. Example cluster with dual-role nodes and dedicated broker nodes 2.1. KRaft limitations KRaft limitations primarily relate to controller scaling, which impacts cluster operations. 2.1.1. Controller scaling KRaft mode supports two types of controller quorums: Static controller quorums In this mode, the number of controllers is fixed, and scaling requires downtime. Dynamic controller quorums This mode enables dynamic scaling of controllers without downtime. New controllers join as observers, replicate the metadata log, and eventually become eligible to join the quorum. If a controller being removed is the active controller, it will step down from the quorum only after the new quorum is confirmed. Scaling is useful not only for adding or removing controllers, but supports the following operations: Renaming a node pool, which involves adding a new node pool with the desired name and deleting the old one. Changing non-JBOD storage, which requires creating a new node pool with the updated storage configuration and removing the old one. Dynamic controller quorums provide the flexibility to make these operations significantly easier to perform. 2.1.2. Limitations with static controller quorums Migration between static and dynamic controller quorums is not currently supported by Apache Kafka, though it is expected to be introduced in a future release. All pre-existing KRaft-based Apache Kafka clusters that use static controller quorums must continue using them. To ensure compatibility with existing KRaft-based clusters, Streams for Apache Kafka continues to use static controller quorums as well. This limitation means dynamic scaling of controller quorums cannot be used to support the following: Adding or removing node pools with controller roles Adding the controller role to an existing node pool Removing the controller role from an existing node pool Scaling a node pool with the controller role Renaming a node pool with the controller role Static controller quorums also limit operations that require scaling. For example, changing the storage type for a node pool with a controller role is not possible because it involves scaling the controller quorum. For non-JBOD storage, creating a new node pool with the desired storage type, adding it to the cluster, and removing the old one would require scaling operations, which are not supported. In some cases, workarounds are possible. For instance, when modifying node pool roles to combine controller and broker functions, you can add broker roles to controller nodes instead of adding controller roles to broker nodes to avoid controller scaling. However, this approach would require reassigning more data, which may temporarily affect cluster performance. Once migration is possible, Streams for Apache Kafka plans to assess introducing support for dynamic quorums. 2.2. Migrating to KRaft mode If you are using ZooKeeper for metadata management in your Kafka cluster, you can migrate to using Kafka in KRaft mode. During the migration, you install a quorum of controller nodes as a node pool, which replaces ZooKeeper for management of your cluster. You enable KRaft migration in the cluster configuration by applying the strimzi.io/kraft="migration" annotation. After the migration is complete, you switch the brokers to using KRaft and the controllers out of migration mode using the strimzi.io/kraft="enabled" annotation. Before starting the migration, verify that your environment can support Kafka in KRaft mode, as there are a number of limitations . Note also, the following: Migration is only supported on dedicated controller nodes, not on nodes with dual roles as brokers and controllers. Throughout the migration process, ZooKeeper and controller nodes operate in parallel for a period, requiring sufficient compute resources in the cluster. Once KRaft mode is enabled, rollback to ZooKeeper is not possible. Consider this carefully before proceeding with the migration. Prerequisites You must be using Streams for Apache Kafka 2.7 or newer with Kafka 3.7.0 or newer. If you are using an earlier version of Streams for Apache Kafka or Apache Kafka, upgrade before migrating to KRaft mode. The Cluster Operator that manages the Kafka cluster is running. The Kafka cluster deployment uses Kafka node pools. If your ZooKeeper-based cluster is already using node pools, it is ready to migrate. If not, you can migrate the cluster to use node pools . To migrate when the cluster is not using node pools, brokers must be contained in a KafkaNodePool resource configuration that is assigned a broker role and has the name kafka . Support for node pools is enabled in the Kafka resource configuration using the strimzi.io/node-pools: enabled annotation. Important Using a single controller with ephemeral storage for migrating to KRaft will not work. During the migration, controller restart will cause loss of metadata synced from ZooKeeper (such as topics and ACLs). In general, migrating an ephemeral-based ZooKeeper cluster to KRaft is not recommended. In this procedure, the Kafka cluster name is my-cluster , which is located in the my-project namespace. The name of the controller node pool created is controller . The node pool for the brokers is called kafka . Procedure For the Kafka cluster, create a node pool with a controller role. The node pool adds a quorum of controller nodes to the cluster. Example configuration for a controller node pool apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaNodePool metadata: name: controller labels: strimzi.io/cluster: my-cluster spec: replicas: 3 roles: - controller storage: type: jbod volumes: - id: 0 type: persistent-claim size: 20Gi deleteClaim: false resources: requests: memory: 64Gi cpu: "8" limits: memory: 64Gi cpu: "12" Note For the migration, you cannot use a node pool of nodes that share the broker and controller roles. Apply the new KafkaNodePool resource to create the controllers. Errors related to using controllers in a ZooKeeper-based environment are expected in the Cluster Operator logs. The errors can block reconciliation. To prevent this, perform the step immediately. Enable KRaft migration in the Kafka resource by setting the strimzi.io/kraft annotation to migration : oc annotate kafka my-cluster strimzi.io/kraft="migration" --overwrite Enabling KRaft migration apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster namespace: my-project annotations: strimzi.io/kraft: migration # ... Applying the annotation to the Kafka resource configuration starts the migration. Check the controllers have started and the brokers have rolled: oc get pods -n my-project Output shows nodes in broker and controller node pools NAME READY STATUS RESTARTS my-cluster-kafka-0 1/1 Running 0 my-cluster-kafka-1 1/1 Running 0 my-cluster-kafka-2 1/1 Running 0 my-cluster-controller-3 1/1 Running 0 my-cluster-controller-4 1/1 Running 0 my-cluster-controller-5 1/1 Running 0 # ... Check the status of the migration: oc get kafka my-cluster -n my-project -w Updates to the metadata state NAME ... METADATA STATE my-cluster ... Zookeeper my-cluster ... KRaftMigration my-cluster ... KRaftDualWriting my-cluster ... KRaftPostMigration METADATA STATE shows the mechanism used to manage Kafka metadata and coordinate operations. At the start of the migration this is ZooKeeper . ZooKeeper is the initial state when metadata is only stored in ZooKeeper. KRaftMigration is the state when the migration is in progress. The flag to enable ZooKeeper to KRaft migration ( zookeeper.metadata.migration.enable ) is added to the brokers and they are rolled to register with the controllers. The migration can take some time at this point depending on the number of topics and partitions in the cluster. KRaftDualWriting is the state when the Kafka cluster is working as a KRaft cluster, but metadata are being stored in both Kafka and ZooKeeper. Brokers are rolled a second time to remove the flag to enable migration. KRaftPostMigration is the state when KRaft mode is enabled for brokers. Metadata are still being stored in both Kafka and ZooKeeper. The migration status is also represented in the status.kafkaMetadataState property of the Kafka resource. Warning You can roll back to using ZooKeeper from this point . The step is to enable KRaft. Rollback cannot be performed after enabling KRaft. When the metadata state has reached KRaftPostMigration , enable KRaft in the Kafka resource configuration by setting the strimzi.io/kraft annotation to enabled : oc annotate kafka my-cluster strimzi.io/kraft="enabled" --overwrite Enabling KRaft migration apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster namespace: my-project annotations: strimzi.io/kraft: enabled # ... Check the status of the move to full KRaft mode: oc get kafka my-cluster -n my-project -w Updates to the metadata state NAME ... METADATA STATE my-cluster ... Zookeeper my-cluster ... KRaftMigration my-cluster ... KRaftDualWriting my-cluster ... KRaftPostMigration my-cluster ... PreKRaft my-cluster ... KRaft PreKRaft is the state when all ZooKeeper-related resources have been automatically deleted. KRaft is the final state (after the controllers have rolled) when the KRaft migration is finalized. Note Depending on how deleteClaim is configured for ZooKeeper, its Persistent Volume Claims (PVCs) and persistent volumes (PVs) may not be deleted. deleteClaim specifies whether the PVC is deleted when the cluster is uninstalled. The default is false . Remove any ZooKeeper-related configuration from the Kafka resource. Remove the following section: spec.zookeeper If present, you can also remove the following options from the .spec.kafka.config section: log.message.format.version inter.broker.protocol.version Removing log.message.format.version and inter.broker.protocol.version causes the brokers and controllers to roll again. Removing ZooKeeper properties removes any warning messages related to ZooKeeper configuration being present in a KRaft-operated cluster. 2.2.1. Performing a rollback on the migration Before the migration is finalized by enabling KRaft in the Kafka resource, and the state has moved to the KRaft state, you can perform a rollback operation as follows: Apply the strimzi.io/kraft="rollback" annotation to the Kafka resource to roll back the brokers. oc annotate kafka my-cluster strimzi.io/kraft="rollback" --overwrite Rolling back KRaft migration apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster namespace: my-project annotations: strimzi.io/kraft: rollback # ... The migration process must be in the KRaftPostMigration state to do this. The brokers are rolled back so that they can be connected to ZooKeeper again and the state returns to KRaftDualWriting . Delete the controllers node pool: oc delete KafkaNodePool controller -n my-project Apply the strimzi.io/kraft="disabled" annotation to the Kafka resource to return the metadata state to ZooKeeper . oc annotate kafka my-cluster strimzi.io/kraft="disabled" --overwrite Switching back to using ZooKeeper apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster namespace: my-project annotations: strimzi.io/kraft: disabled # ...

Commands:
apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaNodePool metadata: name: controller labels: strimzi.io/cluster: my-cluster spec: replicas: 3 roles: - controller storage: type: jbod volumes: - id: 0 type: persistent-claim size: 20Gi deleteClaim: false resources: requests: memory: 64Gi cpu: "8" limits: memory: 64Gi cpu: "12"
annotate kafka my-cluster strimzi.io/kraft="migration" --overwrite
apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster namespace: my-project annotations: strimzi.io/kraft: migration
get pods -n my-project
NAME READY STATUS RESTARTS my-cluster-kafka-0 1/1 Running 0 my-cluster-kafka-1 1/1 Running 0 my-cluster-kafka-2 1/1 Running 0 my-cluster-controller-3 1/1 Running 0 my-cluster-controller-4 1/1 Running 0 my-cluster-controller-5 1/1 Running 0
get kafka my-cluster -n my-project -w
NAME ... METADATA STATE my-cluster ... Zookeeper my-cluster ... KRaftMigration my-cluster ... KRaftDualWriting my-cluster ... KRaftPostMigration
annotate kafka my-cluster strimzi.io/kraft="enabled" --overwrite
apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster namespace: my-project annotations: strimzi.io/kraft: enabled
get kafka my-cluster -n my-project -w
NAME ... METADATA STATE my-cluster ... Zookeeper my-cluster ... KRaftMigration my-cluster ... KRaftDualWriting my-cluster ... KRaftPostMigration my-cluster ... PreKRaft my-cluster ... KRaft
annotate kafka my-cluster strimzi.io/kraft="rollback" --overwrite
apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster namespace: my-project annotations: strimzi.io/kraft: rollback
delete KafkaNodePool controller -n my-project
annotate kafka my-cluster strimzi.io/kraft="disabled" --overwrite
apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster namespace: my-project annotations: strimzi.io/kraft: disabled