## 4 Software
### Components
#### 4.1.4 Apache Kafka  
Apache Kafka is an open-source platform, developed in Java and Scala, for distributed streaming processing. It was originally developed by LinkedIn in order to solve the processing of large amounts of data with low latency, as well as deal with the processing of data in real-time [29]. It was later donated to Apache Software Foundation, its current owner [30].  
Apache Kafka is an event streaming platform, aiming to tackle the challenges in implementing publish/subscribing approaches. With its durable and distributed file system, it is able to support millions of events per second, handling data producers and consumers. Kafka's flexibility and performance enables multiple use cases, from distributing streaming and metrics logging, to the development of event-driven applications, thus being a key component for a data-driven ecosystem [12; 31].  
This platform runs as a cluster and can contain multiple data centers, allowing efficient communication between data producers and consumers using message-based topics. Each message is made up of its key and value, and a date/time stamp [29; 32].  
Kafka constitute of four main APIs, the Producer API, the Consumer API, the Streams API, and the Connector API. While the Producer API enables applications to publish messages to Kafka topics, the Consumer API enables the subscription to one or more topics. The Streams API allows the processing of data in Kafka using stream processing paradigms. These two APIs make subscribers able to process the received messages with stream procedures, then sending to other topics. Finally, the Connector API connects applications or data systems to Kafka topics, providing flexibility in building and managing producers and consumers, as well as providing reusable connections between them [33; 34].  
In Figure 5, the basic architecture of Kafka can be observed.  
Figure 4: Internal architecture of solace messaging. Adapted from Reference [26].  
Components Apache Kafka offers eight different components: Topics, Producers, Consumers, Brokers, Partitions, Replicas, Leaders, and Followers. Hence, the concepts of each of them are presented [31; 32]:  
* **Topics**: A topic is a category where records are published. Topics can be compared to database tables, where multiple subscribers can subscribe to the same topic [31; 32].
* **Producers**: Producers create new messages, and publish them to a specific topic. They are able to assign which partition within the topic the message will go to, by using a message key [31; 32].
* **Consumers**: Consumers label themselves with a consumer group name, and each record published to a topic is delivered to one consumer instance within each subscribing consumer group [31; 32].
* **Brokers**: A Kafka server is called a Kafka broker, receiving messages from producers, assigning offsets to them, and committing the messages to storage. Kafka can work as a cluster, employing multiple brokers to distribute and replicate the messages [31; 32].
* **Partitions**: A partition can be described as an \"ordered, immutable sequence of records that is continually appended to a structured commit log\" [12]. The Kafka cluster will divide topics into partitions and replicate them to brokers [31; 32].
* **Replicas**: Topic replication is essential to design resilient and highly available Kafka deployments [31; 32].
* **Leaders**: Each partition is owned by a single broker, making sure that the follower partitions keep their records synchronized [31; 32].
* **Followers**: Followers serve as replicas for a partition, for, in the event that the leader disconnects, a follower is then promoted to leader [31; 32].  
Figure 6 showcases a perspective of the relationships between these components [33].  
Figure 5: Apache Kafka basic architecture.