Chapter 17. Using Cruise Control to reassign partitions on JBOD disks If you are using JBOD storage and have Cruise Control installed with Strimzi, you can reassign partitions and move data between the JBOD disks used for storage on the same broker. This capability also allows you to remove JBOD disks without data loss. Use the Kafka kafka-log-dirs.sh tool to check information about Kafka topic partitions and their location on brokers before and after moving them. Make requests to the remove_disks endpoint of the Cruise Control REST API to demote a disk in the cluster and reassign its partitions to other disk volumes. Prerequisites You are logged in to Red Hat Enterprise Linux as the Kafka user. You have configured Cruise Control . You have deployed the Cruise Control Metrics Reporter . Kafka brokers use JBOD storage. More than one JBOD disk must be configured on the broker. In this procedure, we use a broker configured with three JBOD volumes and a topic replication factor of three. Example broker configuration with JBOD storage node.id=1 process.roles=broker default.replication.factor = 3 log.dirs = /var/lib/kafka/data-0,/var/lib/kafka/data-1,/var/lib/kafka/data-2 # ... In the procedure, we reassign partitions for broker 1 from volume 0 to volumes 1 and 2. Procedure Start the Cruise Control server. The server starts on port 9092 by default; optionally, specify a different port. cd ./cruise-control/ ./kafka-cruise-control-start.sh config/cruisecontrol.properties <port_number> To verify that Cruise Control is running, send a GET request to the /state endpoint of the Cruise Control server: curl -X GET 'http://<cc_host>:<cc_port>/kafkacruisecontrol/state' (Optional) Check the partition replica data on the broker by using the Kafka kafka-log-dirs.sh tool: kafka-log-dirs.sh --describe --bootstrap-server my-cluster-kafka-bootstrap:9092 --broker-list 1 --topic-list my-topic The tool returns topic information for each log directory. In this example, we are restricting the information to my-topic to show the steps against a single topic. The JBOD volumes used for log directories are mounted at /var/lib/kafka/<volume_id> . Example output data for each log directory { "brokers": [ { "broker": 1, 1 "logDirs": [ { "partitions": [ 2 { "partition": "my-topic-0", "size": 0, "offsetLag": 0, "isFuture": false } ], "error": null, 3 "logDir": "/var/lib/kafka/data-0" 4 }, { "partitions": [ { "partition": "my-topic-1", "size": 0, "offsetLag": 0, "isFuture": false } ], "error": null, "logDir": "/var/lib/kafka/data-1" }, { "partitions": [ { "partition": "my-topic-2", "size": 0, "offsetLag": 0, "isFuture": false } ], "error": null, "logDir": "/var/lib/kafka/data-2" } ] } ] } 1 The broker ID. 2 Partition details: name, size, offset lag. The ( isFuture ) property indicates that the partition is moving between log directories when showing as true . 3 If error is not null , there is an issue with the disk hosting the log directory. 4 The path and name of the log directory. Remove the volume from the node: curl -v -X POST 'http://<cc_host>:<cc_port>/kafkacruisecontrol/remove_disks?dryrun=false&brokerid_and_logdirs=1-/var/lib/kafka/data-0' The command specifies the broker ID and log directory for the volume being removed. If successful, partitions are reassigned from volume 0 on broker 1. Note If required, you can perform a dry run of this operation before applying the changes. Use the Kafka kafka-log-dirs.sh tool again to verify volume removal and data movement. In this example, volume 0 has been removed and my-topic-0 partition reassigned to /var/lib/kafka/data-1 . Example output data following reassignment of partitions { "brokers": [ { "broker": 1, "logDirs": [ { "partitions": [ { "partition": "my-topic-0", "size": 0, "offsetLag": 0, "isFuture": false }, { "partition": "my-topic-1", "size": 0, "offsetLag": 0, "isFuture": false } ], "error": null, "logDir": "/var/lib/kafka/data-1" }, { "partitions": [ { "partition": "my-topic-2", "size": 0, "offsetLag": 0, "isFuture": false } ], "error": null, "logDir": "/var/lib/kafka/data-2" } ] } ] }

Commands:
node.id=1 process.roles=broker default.replication.factor = 3 log.dirs = /var/lib/kafka/data-0,/var/lib/kafka/data-1,/var/lib/kafka/data-2
cd ./cruise-control/ ./kafka-cruise-control-start.sh config/cruisecontrol.properties <port_number>
curl -X GET 'http://<cc_host>:<cc_port>/kafkacruisecontrol/state'
kafka-log-dirs.sh --describe --bootstrap-server my-cluster-kafka-bootstrap:9092 --broker-list 1 --topic-list my-topic
{ "brokers": [ { "broker": 1, 1 "logDirs": [ { "partitions": [ 2 { "partition": "my-topic-0", "size": 0, "offsetLag": 0, "isFuture": false } ], "error": null, 3 "logDir": "/var/lib/kafka/data-0" 4 }, { "partitions": [ { "partition": "my-topic-1", "size": 0, "offsetLag": 0, "isFuture": false } ], "error": null, "logDir": "/var/lib/kafka/data-1" }, { "partitions": [ { "partition": "my-topic-2", "size": 0, "offsetLag": 0, "isFuture": false } ], "error": null, "logDir": "/var/lib/kafka/data-2" } ] } ] }
curl -v -X POST 'http://<cc_host>:<cc_port>/kafkacruisecontrol/remove_disks?dryrun=false&brokerid_and_logdirs=1-/var/lib/kafka/data-0'
{ "brokers": [ { "broker": 1, "logDirs": [ { "partitions": [ { "partition": "my-topic-0", "size": 0, "offsetLag": 0, "isFuture": false }, { "partition": "my-topic-1", "size": 0, "offsetLag": 0, "isFuture": false } ], "error": null, "logDir": "/var/lib/kafka/data-1" }, { "partitions": [ { "partition": "my-topic-2", "size": 0, "offsetLag": 0, "isFuture": false } ], "error": null, "logDir": "/var/lib/kafka/data-2" } ] } ] }