What are the main components of Apache Kafka?
Apache Kafka is a distributed system that is composed of several components. The main components of Apache Kafka are:
- Topics: A topic is a category or feed name to which producers can send messages and read by consumers. It’s append-only and immutable. Consumers can use an offset to track read messages.
- Brokers: A Kafka cluster is composed of one or more brokers that manage the storage of messages in topics and serve as intermediaries between producers and consumers.
- Producers: Applications that produce data and publish it to one or more topics in a Kafka cluster. They control how events are assigned to partitions within a topic.
- Consumers: Applications that subscribe to one or more topics and consume published messages. The only metadata retained on a per-consumer basis is the offset or position of that consumer in a topic. This offset is controlled by the consumer.
- Partitions: Kafka uses partitions to split a topic's messages across multiple brokers for scalability and performance. Events with the same event key, such as the same customer identifier or vehicle ID, are written to the same partition, and Kafka guarantees that any consumer of a given topic partition will always read that partition’s events in exactly the same order as they were written.
- Replication: Multiple brokers always have a copy of the data to guarantee high availability and fault tolerance. This replication is performed at the topic partition level.