Multi-broker Kafka as a service: Run multi-node Kafka cluster on a single machine instantly
By: Hari Krishna
Tags: Docker, Multinode Cluster, Docker Compose, Kafka

Multi-broker Kafka as a service: Run multi-node Kafka cluster on a single machine instantly

multi-broker kafka as docker containers on a docker host

This blog demonstrates a simple utility that I have built which spawns a multi-node Kafka cluster on a single machine via Docker with minimal user intervention. With a basic knowledge of Kafka and Docker; you can make the most of it.

A brief introduction to Apache Kafka.

Kafka is a distributed system consisting of servers and clients that communicate via a high-performance TCP network protocol. It can be deployed on bare-metal hardware, virtual machines, and containers in on-premise as well as cloud environments.

The most interesting observation to be made here is that Kafka can run almost virtually anywhere, on a single node or a cluster of nodes. With the advent of docker and container ecosystem, it is easy to simulate a cluster like environment on a single host with each container isolated from each other with its own namespace, its own volumes. Running a single Kafka broker on a single host is as easy as it gets , just start the Zookeeper server followed by Kafka server.

But the temptation of running multi-node Kafka cluster on a single host or a developer machine(which are predominantly Windows/Mac) is something worth trying :) and a multi-container Kafka setup orchestrated via docker-compose is a natural choice.

Under this Github location is a ready to go 3 node cluster bundled with Kafka 2.5.1, built on Docker version reflecting Docker-compose file format version 3+ ( Refer Docker-compose-versioning to determine the mapping between docker engine version to docker-compose file version).

PFA, the project’s root structure

sw_packages — Packages Kafka 2.5.1 version built against scala 2.12 with few additional helper shell scripts. docker-compose.yml — YAML file defining Zookeeper and Kafka service, the volumes used for persisting Kafka and Zookeeper data, the environment variables required at the containers, network on which the service containers would run. kafka.common.env — Environment file containing a list of environment variables common to all containers. kafka.node0*.env — Collection of environment files with each file dedicated to one Kafka broker node in the cluster. kafka_dockerfile — Dockerfile to build Kafka image required to spawn Kafka containers. startup.sh — Script to bring up Kafka and Zookeeper cluster container services , no additional configuration or parameters required :) shutdown.sh — Script to gracefully shutdown the Zookeeper and Kafka cluster container services. zookeeper_dockerfile — Dockerfile to build Zookeeper image required to spawn Zookeeper containers. zookeeper_override.properties — Configuration to run the Zookeeper ensemble.

Starting up the service:

Executing startup.sh would bring up a 3 node Kafka cluster running on 3 containers with hostname kfk_node01, kfk_node02, kfk_node03 on port 9092 in each nodes and a 3 node Zookeeper cluster running on 3 containers with hostnames zk_node01, zk_node02, zk_node03 running on port 2181 respectively. Running docker-compose ps from the root directory would list all 6 containers running on the same overlay network which in a nutshell is the entire Zookeeper-Kafka ensemble.

The network type used is Overlay. This allows an external container to attach itself to it and once the services come up an external container with libraries packaging Kafka-client could be used to create topics on the Kafka cluster , produce messages to topics , consume messages from it. For instance, docker image krishari2006/kafka:v2.5.1 from DockerHub could be used to spin up a container with working directory set to {KAFKA_HOME}/bin folder.

docker run -it --network kafka_251_service_kafka_net krishari2006/kafka:v2.5.1 /bin/bash

Networks created by docker-compose are prefixed with the root/parent directory’s name followed by the network name defined in the docker-compose YAML file, kafka_net in this case and running docker network ls | grep ${Parent Directory} lists the network which connects all containers together. A Host network type could not be used as it is still not supported in Docker on Mac/Windows.

Shutting down the service:

shutdown.sh would gracefully shutdown all the containers. Each Kafka broker container is backed by persistent docker volume so that Kafka topic data is not lost between service restarts.

Running Docker-compose down command is not encouraged since it sends a SIGTERM first and if it is not successful then a SIGKILL to forcefully kill all container process. This corrupts the Kafka broker’s state, an inconsistent state with the Zookeeper service between service restarts

ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)

kafka_1 | org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists

kafka_1 | at org.apache.zookeeper.KeeperException.create(KeeperException.java:122)

shutdown.sh script’s true purpose is to gracefully orchestrate de-initialization process , by shutting down Kafka brokers followed by Zookeeper servers.

Beyond the defaults:

Even though the intent of this utility is to rapidly spawn a 3 node multi-broker Kafka service on a single docker host it could be customized by modifying very small collection of files. Let us discuss about properties in each of these files which are essential for horizontal scale up/down of the service.

kafka.common.env : To scale up or scale down the Zookeeper nodes kafka_zookeeper_connect entry needs to be modified which is a comma-separated list of Zookeeper node:port value

kafka_zookeeper_connect=zk_node01:2181,zk_node02:2181,zk_node03:2181

2. kafka.node{#node_num}.env: Contains Kafka broker specific properties. #node_num represents Kafka node’s ${broker.id} property +1. To add an additional node to the Kafka ensemble, ensure following properties are set

brkr_id — An integer that represents the broker.id property. It has to be unique and should not conflict with existing broker’s id.

brkr_id=0

kafka_listeners — Represents listeners property of the broker. The current template/pattern exposes two listeners , one internal and other external listening at 9091 and 9092 respectively bound to all network interfaces. External listener is the one associated to advertised listener to interface with other brokers in the ensemble and also clients.

kafka_listeners=INTERNAL://:9091,EXTERNAL://:9092

kafka_advertised_listeners — Represents advertised.listeners property of the Kafka broker. As mentioned earlier EXTERNAL listener would be the listener type.

kafka_advertised_listeners=EXTERNAL://kfk_node01:9092

3.zookeeper_override.properties:

dataDir=/tmp/zookeeper

# the port at which the clients will connect

clientPort=2181

# disable the per-ip limit on the number of connections since this is a non-production config

maxClientCnxns=0

# Disable the adminserver by default to avoid port conflicts.

# Set the port to something non-conflicting if choosing to enable this

admin.enableServer=false

# admin.serverPort=8080

initLimit=5

syncLimit=2

server.1=zk_node01:2888:3888

server.2=zk_node02:2888:3888

server.3=zk_node03:2888:3888

dataDir — The location on the docker container used by Zookeeper service to store Zookeeper File System’s nodes(zk nodes). This is crucial while defining volumes for the containers as this would be the mount location inside the container to persist Zookeeper data on the host volume.

clientPort — Port to be used by the clients. In this case the Kafka broker would use these ports to back its metadata on to the Zookeeper nodes (by default it is 2181). Any change to this value should be reflected in kafka_zookeeper_connect property in kafka.common.env file.

server.${#node_num} — By default, there are three entries, each representing a Zookeeper node and each value is dissected into three parts -first, the hostname followed by port (by default 2888) to communicate with its peer nodes followed by another port ( by default 3888) used during leader election, all separated from each other by colon.

In spite of all safe start-up and shutdown mechanisms, it is possible for services to end up in stale state. This could be rectified by deleting the volumes used by each service container.

docker container prune --force docker volume rm --force kafka_251_service_kfk_node01_vol kafka_251_service_kfk_node02_vol kafka_251_service_kfk_node03_vol kafka_251_service_zk_node01_vol kafka_251_service_zk_node02_vol kafka_251_service_zk_node03_vol

And we have reached the end of it. The source code is available under Github location. The idea could be extended to bundle any version of Kafka with few minor tweaks and the end result would just be the same. Feedbacks, if any, are appreciated 😊.