Streaming system tutorial with Flink and Kafka
By: Bala Dutt
Tags: Flink, Kappa, Flink Streaming, Kafka, Streaming

Bala Dutt, Charu Garg, Prabhat Hegde

A Simple Kappa architecture

Streaming enables the obvious: continuous processing of data that is continuously being produced.

Having read enough about Kafka (vs Lambda or Omega) architectures, it is now time to get hands dirty. Let us build a simple streaming system. It is extremely easy to build this with current state of art. We intend to stay at a level that gives us conceptual picture along with focus on business usecase and solution.

Architecture

The 10,000 ft view is we will connect Kafka through Flink. We will push messages into Kafka and Flink would feed those to stream. This is essentially a Kappa architecture.

ZooKeeper is a centralised service for maintaining configuration information, naming, providing distributed synchronisation, and providing group services. ZooKeeper is used by many distributed systems. It serves as the coordination interface between the Kafka brokers and consumers. Kafka stores basic metadata in Zookeeper such as information about topics, brokers, consumer offsets (queue readers) and so on. The leader election between the Kafka broker is also done by using Zookeeper in the event of leader failure.

Kafka allows producers to publish messages and consumers to read. It is simply a collection of topics split into one or more partitions. Each partition is a linearly ordered sequence of messages, where each message is identified by their index (called as offset). Incoming messages are written at the end of a partition and messages are sequentially read by consumers.

Flink is a true stream processing framework. Flink’s low latency, high throughput architecture is a great fit for the time-sensitive and real-time elements of products. Unbounded stream of events can be processed at scale using processing functions or continuous operators implemented in Java and submitted to Flink.

How the setup looks like

Setup

Follow instructions in the project (github).

Setup ZooKeeper with few basic properties,

tickTime : the basic time unit in milliseconds used by ZooKeeper. It is used to do heartbeats and the minimum session timeout will be twice the tickTime.

: the basic time unit in milliseconds used by ZooKeeper. It is used to do heartbeats and the minimum session timeout will be twice the tickTime. dataDir : the location to store the in-memory database snapshots and, unless specified otherwise, the transaction log of updates to the database.

: the location to store the in-memory database snapshots and, unless specified otherwise, the transaction log of updates to the database. clientPort : the port to listen for client connections. This is usually 2181.

Setup Kafka as mentioned.

Flink will be downloaded as part of maven build.

Code

com.stream has two usecases that are isolated. Focus on simpler usecase : Telecom. Here we process telecom records to find location or user based lower or higher usage and generate alerts. The alerts could be consumed by operator to learn of issues in the system or by user to be notified of usage. Below is a picture of usecase,

A simple telecom usecase

The system works as follows,

An event generator will simulate call events and act as call-data source. Apache Kafka will be used to publish these events. Apache Flink with Kafka as source will be used as Stream processing f/w Call Record will have Phone numbers, Call Origin, Call Destination, Call Start Time and duration of call Transformations will occur in Flink to serve 5 provisioning alerts as given below. Alert 1 — Low and High Threshold Alerts based on calls originating from a location (Window based aggregation used to accomplish it). Alert 2 — Low and High Threshold Alerts based on calls terminating to a location (Window based aggregation used to accomplish it). Alert 3 — Low and High Threshold Alerts based on calls originating from and terminating to a location. Alert 4, Alert 5: The same analysis is done on a per-user basis. A Notification System can then process the alerts. These alerts can be used by the Telco’s provisioning system to make changes.

The core of processing logic is in TelecomUsageWorkflow class.

Running it

There are three steps,

Start ZooKeeper and Kafka

Build the streaming pipeline and run it using Flink. This is done by building the project

Generate events using a simulator

And then output may be checked in “alerts*” folders and as also on stdout.

A note on scale

So, how does one have a simple conceptual model of scale in this. First, Kafka scales by partitioning a topic. So, choice of key is important. Note, partition is lowest unit in Kafka. Then, brokers may be scaled, even multiple topics too may help. Kafka delegates responsibility of remembering what they have seen to clients, so Kafka doesn’t need to know details of all the consumers.

Flink scales by creating multiple instances of operators on multiple nodes. Unless the workflow needs a partition by a key and a window, then the key and length of window becomes important.

If there’s a logic problem in operators, they may be redeployed and messages be replayed from Kafka. So, Kafka is store for messages for the duration needed.

Conclusion

It is easy to build a streaming application. A high-level simple view of components allows one to stay focussed on business usecase and the solution. Each of the components though has lot of depth available to explore and master. These and the benefits of this architecture will be subject of future articles.