Kafka For Beginners — Part 1. Kafka is a transformative tool, and a…
By: Bawya Pandian
Tags: Kafka Part1, Kafka, Kafka Streams, Kafka Connect, Kafka Beginners

Kafka is a transformative tool, and a key component to many event-based systems today.

A publish-subscribe (pub-sub) message system that allows messages (also called records) to be sent between processes, applications, and servers. Simply said - Kafka stores streams of records. Conceptually, we can think of Kafka as comprising three basic components:

An Event Log , to which messages are published.

, to which messages are published. Publishers , which publish messages to the event log.

, which publish messages to the event log. Consumers, which consume messages from the event log record can include any kind of information.

typical message queue and its consumers work

For e.g. have information about an event that has happened on a website or could be a simple text message that triggers an event so another application may connect to the system and process or reprocess it.

A Kafka cluster consists of a chosen number of brokers/servers (also called nodes). Apache Kafka itself is storing streams of records. A record is data containing a key, value and timestamp sent from a producer. The producer publishes records on one or more topics. You can think of a topic as a category to where, applications can add, process and reprocess records (data). Consumers can then subscribe to one or more topics and process the stream of records.

Kafka is often used when building applications and systems in need of real-time streaming.

Topics :

All Kafka records are organized into topics. Topics are the categories in the Apache Kafka broker to where records are published. Data within a record can be of various types, such as String or JSON. The records are written to a specific topic by a producer and subscribed from a specific topic by a consumer.

Partition:

Kafka topics are divided into partitions.

Partitions allow you to parallelize a topic by splitting the data into a topic across multiple nodes.

Each record in a partition is assigned and identified by its unique offset​. ​This offset points to the record in a partition. Incoming records are appended at the end of a partition. The consumer then maintains the offset to keep track of the next record to read. Kafka can maintain durability by replicating the messages to different brokers (nodes).

A topic can have multiple partitions. This allows multiple consumers to read from a topic in parallel. The producer decides which topic and partition the message should be placed on.

Consumers and consumer groups

Consumers can read messages starting from a specific offset and are allowed to read from any offset point they choose. This allows consumers to join the broker at any point in time.

Each of those consumers would then belong to a single consumer group. The purpose of the consumer group is to ensure that all messages from a topic are read and processed by the application, while allowing this processing to be done in parallel.

Replication — the power of copying and reproducing

In Kafka, replication is implemented at the partition level. The redundant unit of a topic partition is called a replica. A follower that is in-sync is called an in-sync replica. If a partition leader fails, a new in-sync replica is selected as the new leader. Each partition usually has one or more replicas meaning that partitions contain records that are replicated over a chosen number of Kafka brokers in the cluster.

It’s possible for the producer to attach a key to the records and tell which partition the record should go to. These keys can be useful if you wish for a strong order, in case you are developing something that requires, for example, a unique id. When attaching a key to these records, it will ensure that records with the same key will arrive at the same partition​.

Reference: