Getting started with Apache Kafka
By: Ankita Gulati
Tags: Kafka, Zookeeper, Event Driven Architecture, Getting Started, Pub Sub

Engineering a Resilient Messaging Service (Part -2)

I have been working with one of the leading companies in travel domain from last 3.5 years as a software developer and when it comes to communication between the micro services, I have always heard one name — KAFKA

When I read about Kafka, I came to know that Kafka is a messaging service and it’s not just my company but most of the organisations are using this intensively. This, actually made me curious to understand, what exactly a simple messaging service is providing that most of the companies are relying on this. So, I decided to dive deep into understanding Kafka.

Kafka is a distributed messaging system that lets you send messages between processes, applications, and servers, providing fast, highly scalable and redundant messaging through a pub-sub model.

But do I really need Kafka? Why should I complicate my system with a new technology? Can’t I find or create a simple solution for exchanging data between my micro services? If such question arise in your mind too, I would recommend you to read Part-1 of this blog.

Before discussing the wonders Kafka can do, let’s discuss few key terms that comprises the Kafka system.

Message: Information that is sent from the producer to a consumer through Kafka.

Producer: Producer is an application that sends messages. It does not send messages directly to the recipient. It send messages only to the Kafka server.

Consumer: Consumer is the recipient. It is an application that reads messages from the Kafka server.

Brokers: Brokers are simply servers responsible for maintaining the published data.

Cluster: A cluster is a group of nodes i.e. a group of brokers that makes the Kafka a distributed messaging service

Topic: A topic is a place where all the messages are written. It is just like any database’s table where data is stored.

Partition: A topic is further divided to partitions , which allows you to split data across multiple brokers. One partition can only be on one broker. Producer does not need to worry anything about partitions while pushing data to the Kafka server, it will send data to particular topic and Kafka will manage and push the data to a particular partition.

Offset: The offset is a unique identifier of a record within a partition. It denotes the position of the consumer in the partition.

Replication Factor: A replica of a partition is a “backup” of a partition. Replicas never read or write data. They are used to prevent data loss. The number of replicas required is called the replication factor.

Consumer Group: Since the data is stored in a distributed manner in partitions, the efficiency of the system can be increased if data is consumed in parallel from different partitions. A consumer group includes the set of consumer processes that are subscribing to a specific topic. Each consumer in the group is assigned a set of partitions to consume from. They will receive messages from a different subset of the partitions in the topic. Kafka guarantees that a message is only read by a single consumer in the group. Each partition can be read only by one consumer in a group, so having consumers count greater than partitions won’t give you any performance benefits.

Zookeeper: Kafka system can’t work without a zookeeper. The Zookeeper manages the brokers. It notifies the producers and consumers when a new broker enters the Kafka system or if a broker fails in the system. On being informed about the failure of a broker, the producer and consumer decide how to act and start coordinating with some other broker.

Below is the architectural diagram of Kafka:

Now, we have enough theoretical knowledge, It’s high time to do some hands-on with Kafka.

Install Kafka:

If Kafka is not installed on your system, I would recommend you to download Kafka from this link and start Kafka with the command below:

<path-to-confluent>/bin/confluent local start

Now consider the example discussed in part 1. Service A wants to send data to Service B,C,D. Service A is the producer and rest of the services are the consumers.

Let’s create a topic first, where service A will push the data.

<path-to-confluent>/bin/kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic bookings

Till now, we have created a new topic “bookings” with 1 partition and 1 replication factor.

Time to write a producer and consumer now:

Producer:

<path-to-confluent>/bin/kafka-console-producer --broker-list localhost:9092 --topic bookings

Consumer:

<path-to-confluent>/bin/kafka-console-consumer --bootstrap-server localhost:9092 --topic bookings --group service_b_consumer

Similarly, we will write consumer for service C and D.

Now let’s push some data from producer and see if service B is able to read this data.

Push some data to the bookings topic as shown below:

Producer

Switch tab, and check if your consumers is able to read the messages pushed by the producer. The consumer will look something like below:

Consumer

Well, Mission Accomplished!! But do you really think so?

No doubt, the task is done here, but if you are really curious to understand the internals of Kafka and working behind these commands, It’s time to move ahead to the next part of our blog.