Kafka Connect in a nutshell
By: Maciej Szymczyk
Tags: Kafka, Streaming, Big Data, Apache Kafka, Kafka Connect

Kafka Connect is part of the Apache Kafka platform. It is used to connect Kafka with external services such as file systems and databases. In this story you will learn what problem it solves and how to run it.

Why Kafka Connect?

Apache Kafka is used in microservices architecture, log aggregation, Change data capture (CDC), integration, streaming platform and data acquisition layer to Data Lake. Whatever you use Kafka for, data flows from the source and goes to the sink.

It takes time and knowledge to properly implement a Kafka’s consumer or producer. The point is that the inputs and outputs often repeat themselves. Many companies pull data from Kafka to HDFS/S3 and Elasticsearch. What if we just could use one size fits all implementation?

There is no point in reinventing the wheel. Kafka Connect solves this problem. It is a platform to connect Kafka with external components. We can configure inputs and outputs with connectors. It can run it standalone and distributed mode. As a result we have scalable and fail-tolerant platform at out disposal.

Alternatives

The alternatives that come to my mind are:

Connectors

Connectors can be found on the Confluent Hub website. They are available free of charge as well as paid, available within the Confluent platform. For example: The Cassandra Connector is available in a paid version (from Confluent), but there is also a free version from DataStax. If you have not found a ready-made solution, you can implement connector on your own.

Environment

In this case, Kafka, Zookeeper and Minio will run on Docker. Kafka Connect will run on the host machine. It is part of the Apache Kafka, so just download and unpack the binaries. Conduktor is a pretty handy GUI Kafka tool.

version: '2'



services:

zookeeper:

image: 'bitnami/zookeeper:3'

ports:

- '2181:2181'

volumes:

- 'zookeeper_data:/bitnami'

environment:

- ALLOW_ANONYMOUS_LOGIN=yes



kafka:

image: 'bitnami/kafka:2'

ports:

- '9092:9092'

- '29092:29092'

volumes:

- 'kafka_data:/bitnami'

environment:

- KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181

- ALLOW_PLAINTEXT_LISTENER=yes

- KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,PLAINTEXT_HOST://:29092

- KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT

- KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092

depends_on:

- zookeeper



minio:

image: 'bitnami/minio:latest'

ports:

- '9000:9000'

environment:

- MINIO_ACCESS_KEY=minio-access-key

- MINIO_SECRET_KEY=minio-secret-key





volumes:

zookeeper_data:

driver: local

kafka_data:

driver: local

Standalone mode — writing to file

Apache Connect can be run in two modes: Standalone and Distributed. The first one is for testing and development. To run it you need configuration files.

bootstrap.servers=localhost:29092



key.converter=org.apache.kafka.connect.storage.StringConverter

value.converter=org.apache.kafka.connect.storage.StringConverter



offset.storage.file.filename=/tmp/connect.offsets

The key and value converter define the format in which Kafka reads and writes data.

name=standalone-file-sink

connector.class=FileStreamSink

tasks.max=1

file=output.txt

topics=standalone-test

The fields seem obvious. Below the GIF of the execution.

kafka_2.13-2.6.0/bin/connect-standalone.sh connect-standalone.properties connect-file-sink.properties

Distributed Mode

This mode provides scalability and fault tolerance. You only need to run multiple workers with the same group.id, Task Orchestration is provided by the platform itself. In the DataGen example you will see how Kafka Connect behaves when you kill one of the workers. Below is the configuration file of one of the workers.

bootstrap.servers=localhost:29092

group.id=connect-cluster

key.converter=org.apache.kafka.connect.json.JsonConverter

value.converter=org.apache.kafka.connect.json.JsonConverter



key.converter.schemas.enable=false

value.converter.schemas.enable=false



offset.storage.topic=connect-offsets

offset.storage.replication.factor=1



config.storage.topic=connect-configs

config.storage.replication.factor=1



status.storage.topic=connect-status

status.storage.replication.factor=1



#rest.host.name=

rest.port=8083



plugin.path=/tmp/plugins/

group.id = works like a consumer group. Workers in a cluster must have identical group.id.

Kafka Connect keeps data in… Kafka. You have to indicate the names of the topics and the expected replication factor.

We put the connectors into the directory provided by plugin.path. All workers should have access to the same connectors.

All we have to do is to run Kafka Connect

kafka_2.13-2.6.0/bin/connect-distributed.sh connect-distributed-1.properties

Data generation — Datagen Source

We will use Kafka Connect Datagen. It is a connector for generating data. It can be useful for some PoC. Just download it ,unzip and put into the directory given in plugin.path.

The tasks in Kafka Connect are run using the REST API. First you need to prepare the configuration of the connector.

{

"connector.class": "io.confluent.kafka.connect.datagen.DatagenConnector",

"kafka.topic": "distributed-test",

"quickstart": "ratings",

"key.converter": "org.apache.kafka.connect.storage.StringConverter",

"value.converter": "org.apache.kafka.connect.json.JsonConverter",

"max.interval": 100,

"tasks.max": "1"

}

To then send it to the …/connectors/name/config URL

curl -X PUT -H 'Content-Type:application/json' http://localhost:8083/connectors/source-datagen/config -d @connect-datagen-source.json

But we can also use other tools, e.g. the Conduktor.

Below you can see the GIF in which I run the Datagen connector and then start and stop Kafka Connect workers.

Moving data out — AWS S3 Sink (MinIO)

Data is generated, let’s move on to sending it to other systems. Originally I wanted to use Elasticsearch Sink here, but I couldn’t get around the problem with the lack of guava in Kafka libraries. You can find AWS S3 connector here. The same steps as for Datagen.

{

"name":"sink-minio",

"connector.class":"io.confluent.connect.s3.S3SinkConnector",

"tasks.max":"1",

"topics":"distributed-test",

"s3.bucket.name":"test",

"s3.part.size":"5242880",

"s3.compression.type":"gzip",

"flush.size":"100",

"aws.secret.access.key": "minio-secret-key",

"aws.access.key.id": "minio-access-key",

"store.url":"http://localhost:9000",

"value.converter": "org.apache.kafka.connect.json.JsonConverter",

"format.class": "io.confluent.connect.s3.format.json.JsonFormat",

"storage.class": "io.confluent.connect.s3.storage.S3Storage",

"schema.compatibility": "NONE",

"schemas.enable":false

}

store.url points to a local container with MinIO

flush.size — number of records that will be in one package

s3.compression.type — each package will be compressed (or not)

value.converter — records can be in specific formats. Here we use JSON.

curl -X PUT -H 'Content-Type:application/json' http://localhost:8083/connectors/sink-minio/config -d @connect-minio-sink.json

The MinIO task has been added. We can check if the data actually goes into the bucket.

Summary

If the data is to go straight to/from Kafka, Kafka Connect may be a good choice. For more advanced operations I rather see a solution in streaming climates such as Logstash or Spark Structured Streaming.

Repo

https://github.com/zorteran/wiadro-danych-kafka-connect-nutshell