A practical streaming data infrastructure case with Fluentd, Kafka, Kafka Connect, ElasticSearch, Kibana and Docker
By: Emmanuel Hodonou
Tags: Docker, Elasticsearch, Fluentd, Kafka, Kafka Connect

This is a practical case of setting up a continuous data infrastructure. The stack allows for a distributed log system. Fluentd retrieves logs from different sources and puts them in kafka. Kafka Connect retrieves Kafka data logs for indexing in ElasticSearch.

Our stack is composed of:

Fluentd : An open source data collector for unified logging layer. I used a connector (@type kafka2) to send logs in Kafka. Kafka : Distributed streaming platform. Fluentd produces data in Kafka. Zookeeper : Kafka run with Zookeeper. Zookeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. Kafka Connect : an component of Kafka, is a framework for connecting Kafka with external systems such as databases, key-value stores, search indexes, and file systems. It is used here to easily connect to kafka and send data in Elasticsearch. I used the Kafka Connect ElasticSearch kafka-connect-ui : Used to have access to Kafka Connect UI. I used it to set up connector without go on command line. Elasticsearch : It is an distributed platform and a RESTful search and analytics engine. Used here to index and easily search data. Kibana : UI to connect and view data or logs stored in Elasticsearch. python-app-test : A Python application to test the logs streaming.

What will we see in this article ?

We will start the Apache Kafka service.

We will start an instance of Elasticsearch and Kibana

We will see how to configure the Kafka Connect service to broadcast Kafka data and automatically store it in Elasticsearch.

We will start a fluentd service and configure it to automatically publish logs on a kafka topic.

We will see how to call the fluentd service in a python application to send python logs to Fluentd.

We will set up an index pattern on Kibana to visualize data logs.

Dependencies

Docker installed on your machine

Let’s create our stack folder

$ mkdir agile-logging

$ cd agile-logging

Let’s prepare the necessary docker files

Open your terminal and create the file Dockerfile-fluentd and past this code

FROM fluent/fluentd:v1.6-debian-1

USER root

ENV FLUENT_UID=0

RUN mkdir /buffer

RUN ["gem", "install", "fluent-plugin-kafka", "--no-document", "--version", "0.12.1"]

Create the file Dockerfile-kafka-connect and paste this lines

FROM confluentinc/cp-kafka-connect:4.0.0



RUN apt-get update && apt-get install -y vim



RUN mkdir -p /opt

RUN mkdir -p /opt/connectors

ADD connector_jars /opt/connectors/connector_jars

WORKDIR /opt



ADD connector_conf/ /opt/connector_conf

RUN wget https://github.com/confluentinc/kafka-connect-elasticsearch/archive/v3.3.3-rc1.tar.gz; \

tar xzf v3.3.3-rc1.tar.gz -C /opt/connectors

Create the file Dockerfile-python-app-test.yaml and paste this lines

FROM pypy:3-6



WORKDIR /usr/src/app



# Bundle app source

COPY python-app-test/ /usr/src/app





RUN apt-get update



RUN apt-get install -y supervisor



# install requirements

RUN pip install --upgrade pip

RUN pip install -r requirements.txt



RUN mkdir -p /var/log/supervisor

RUN mkdir -p /usr/src/logs

COPY python-app-test/src/conf/agilecx.conf /etc/supervisor/conf.d/supervisord.conf





EXPOSE 611



VOLUME ["/usr/src/app"]



# run supervisord



CMD /usr/bin/supervisord -c /etc/supervisor/conf.d/supervisord.conf

Let’s prepare our stack of docker compose

We will use a docker compose file to start our stack. Let’s create the docker-compose-dist-logging.yaml file and reference our docker files.

We will use a docker composition file to start our stack. Let’s create the docker-compose-dist-logging.yaml file and refer to our docker files

version: '3'

services:

fluentd-agile:

container_name: fluentd-agile

build:

context: ./

dockerfile: Dockerfile-fluentd

volumes:

- ./fluentd/conf:/fluentd/etc

ports:

- "24228:24224"

- "24228:24224/udp"

networks:

- agile-network zookeeper-agile-node1:

image: zookeeper:3.4.9

hostname: zookeeper-agile-node1

container_name: zookeeper-agile-node1

ports:

- 2190:2181

volumes:

- ../zookeeperkafka-data/vol1/zk-data:/data_logs_dir

- ../zookeeperkafka-data/vol2/zk-txn-logs:/logs_dir

restart: always

environment:

ZOO_MY_ID: 1

ZOO_PORT: 2181

ZOO_SERVERS: server.1=zookeeper-agile-node1:2888:3888

ZOO_LOG4J_PROP: /logs_dir

ZOO_DATA_LOG_DIR: /data_logs_dir

networks:

- agile-network



kafka-agile-node1:

image: confluentinc/cp-kafka:5.1.0

hostname: kafka-agile-node1

container_name: kafka-agile-node1

ports:

- 5656:9092

volumes:

- ../zookeeperkafka-data/vol3/kafka-data:/logs

restart: always

environment:

KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka-agile-node1:9092,LISTENER_DOCKER_EXTERNAL://kafka-agile-node1:5656

KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT

KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL

KAFKA_ZOOKEEPER_CONNECT: "zookeeper-agile-node1:2181"

KAFKA_CREATE_TOPICS: log-messages:1:1,connect-config:1:1:compact,connect-offset:1:1:compact,connect-status:1:1:compact

KAFKA_BROKER_ID: 1

KAFKA_LOG_DIRS: /logs

KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"

KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

KAFKA_REPLICA_FETCH_MAX_BYTES: 2048576

KAFKA_PRODUCER_MAX_REQUEST_SIZE: 2048576

KAFKA_CONSUMER_MAX_PARTITION_FETCH_BYTES: 2048576

depends_on:

- zookeeper-agile-node1

networks:

- agile-network



kafka_connect:

container_name: kafka_connect

build:

context: ./

dockerfile: Dockerfile-kafka-connect

depends_on:

- kafka-agile-node1

- elasticsearch-agile-one

links:

- kafka-agile-node1

- elasticsearch-agile-one

environment:

CONNECT_BOOTSTRAP_SERVERS: kafka-agile-node1:9092

CONNECT_GROUP_ID: 1

CONNECT_CONFIG_STORAGE_TOPIC: connect-config

CONNECT_OFFSET_STORAGE_TOPIC: connect-offset

CONNECT_STATUS_STORAGE_TOPIC: connect-status

CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1

CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1

CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1

CONNECT_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter

CONNECT_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter

CONNECT_KEY_CONVERTER_SCHEMAS_ENABLE: "false"

CONNECT_VALUE_CONVERTER_SCHEMAS_ENABLE: "false"

CONNECT_INTERNAL_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter

CONNECT_INTERNAL_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter

CONNECT_REST_ADVERTISED_HOST_NAME: localhost

CONNECT_PLUGIN_PATH: "/usr/share/java,/opt/connectors"

networks:

- agile-network



kafka-connect-ui:

container_name: kafka-connect-ui

image: landoop/kafka-connect-ui:0.9.7

hostname: kafka-connect-ui

ports:

- "8003:8000"

environment:

CONNECT_URL: "http://kafka_connect:8083/"

PROXY: "true"

depends_on:

- kafka_connect

networks:

- agile-network



elasticsearch-agile-one:

image: docker.elastic.co/elasticsearch/elasticsearch:7.4.0

container_name: elasticsearch-agile-one

restart: always

environment:

- node.name=elasticsearch-agile-one

- cluster.name=docker-agile-cluster

- bootstrap.memory_lock=true

- http.cors.enabled=true

- http.cors.allow-origin=*

- discovery.seed_hosts=elasticsearch-agile-one

- cluster.initial_master_nodes=elasticsearch-agile-one

- cluster.routing.allocation.disk.threshold_enabled=false

- "ES_JAVA_OPTS=-Xms512m -Xmx512m"

ulimits:

nproc: 65535

memlock:

soft: -1

hard: -1

volumes:

- ../elasticsearch-data/elk-one/logs:/var/log

- ../elasticsearch-data/elk-one/data:/usr/share/elasticsearch/data

networks:

- agile-network

ports:

- 9200:9200

- 9300:9300



kibana-agile:

image: 'docker.elastic.co/kibana/kibana:7.4.0'

container_name: kibana-agile

environment:

- ELASTICSEARCH_HOSTS=http://elasticsearch-agile-one:9200

ports:

- '5601:5601'

networks:

- agile-network



python-app-test:

container_name: python-app-test

build:

context: ./

dockerfile: Dockerfile-python-app-test.yaml

ports:

- "7500:611"

volumes:

- ./python-app-test/:/usr/src/app

- ../agilecx-data/logs/:/usr/src/logs

networks:

- agile-network

networks:

agile-network:

Before executing the composition file, we have to proceed to certain configurations, otherwise we will have errors.

To run the fluentd service referenced in our compose file, we need to create a Fluentd-accepted configuration file.

The Fluentd configuration file

Let’s create the fluentd folder in our source folder.

$ mkdir fluentd

Let’s create the conf folder in our fluentd folder.

$ cd fluentd

$ mkdir conf

$ cd conf

We create the file fluent.conf in the conf folder and paste this

# TCP input

<source>

@type forward

port 24224

</source>

<match *.**>

@type kafka2



# list of seed brokers

brokers kafka-agile-node1:9092

use_event_time true



# buffer settings

<buffer topic>

@type file

path /buffer/td

flush_interval 3s

</buffer>



# data type settings

<format>

@type json

</format>



# topic settings

topic_key topic

default_topic log-messages



# producer settings

required_acks -1

compression_codec gzip

</match>

Let’s take a closer look at the fluentd configuration file.

The directives

<source>

@type forward

port 24224

</source>

The source directives determine the input sources and the source plugin forward turns fluentd into a TCP endpoint to accept TCP packets through port 24224. So we receive log events from 24224/tcp

<match *.**>

</match>

The match directives determine the output destinations. We use a Kafka plugin called kafka2. The Kafka2 plugin is built for Fluentd. This plugin is responsible for sending logs events to Kafka every 3 seconds (flush_interval 3s). With the brokers tag, we have referenced our Kafka service name kafka-agile-node1:9092 added to our docker compose file. The Kafka topic name used is log-messages.

Fluentd documentation is quite detailed. I invite you to browse it.

The files used by the service kafka connect

Kafka Connect is an open source component of Kafka, is a framework for connecting Kafka with external systems such as databases, key-value stores, search indexes, and file systems.

Using Kafka Connect you can use existing connector implementations for common data sources and sinks to move data into and out of Kafka.

Here we are using Kafka connect to get logs from Kafka and automatically send these logs in ElasticSearch.

Create the folders connector_conf and connector_jars in the root source folder.

$ mkdir connector_conf

$ mkdir connector_jars

Copy and paste from Github the files of these two folders.

The python example application

We used Falcon framework to build an API. At each call of the API:

1- Logs are sent to Fluentd. Fluentd pluging Kafka2 send data in Kakfa.

2- Kafka Connector read Kafka topic “log-messages” and send logs in ElasticSearch.

Check the source code on Github for the python application.

Run the compose file

Run the docker-compose and wait until services started:

$ docker-compose -f docker-compose-dist-logging.yaml up --build --d

After all services created, check services running:

$ docker ps

You should see the following services unordered running:

python-app-test

fluentd-agile

zookeeper-agile-node1

kafka-agile-node1

kafka_connect

kafka-connect-ui

elasticsearch-agile-one

kibana-agile

At this step data are not sending yet. We need to activate Kafka source connector

Let’s start the Kafka connector plugin

We have 2 possibilities to start connector:

Using the command

Enter in the docker

$ docker exec -it kafka_connect bash

Inside the docker, add the the connector by running the following command :

You will see an output like the following. It means that your connector has been added :

You will see some sort of following output. it means that your connector has been added.

{

{

"name":"elasticsearch-sink",

"config":{"connector.class":"io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",

"tasks.max":"1",

"topics":"log-messages",

"key.ignore":"true",

"schema.ignore":"true",

"connection.url":"http://elasticsearch-agile-one:9200",

"type.name":"kafka-connect",

"name":"elasticsearch-sink"

},

"tasks":[],

"type":null

}

2. Using the Kafka-connect UI

Open the browser and go on http://localhost:8003

Click on the button “New” to add new connector Choose ElasticSearch Connector and fill json like in the file connector_conf/connector_elasticsearch.json

{ "name":"elasticsearch-sink", "config":{ "connector.class":"io.confluent.connect.elasticsearch.ElasticsearchSinkConnector", "tasks.max":"1", "topics":"log-messages", "key.ignore":"true", "schema.ignore": "true", "connection.url": "http://elasticsearch-agile-one:9200", "type.name": "kafka-connect" }

Click on the button “create” when you finish.

Open your browser and go on http://localhost:8003. You should see the count of SINK CONNECTORS at 1. It means connectors has been added.

Data visualization with Kibana

Let’s check now with Kibana if the connector has created our indice in Elasticsearch.

Open your browser and go on http://localhost:5601/

Find “Management” menu and click on it . Under Elasticsearch section, click on “Index Management” and you should see your indice created with the name “log-messages” . To visualize data with Kibana, we should create an index pattern. So under Kibana section, click on “Index Patterns” and click on “Create index pattern”. Fill the index pattern name “log-messages*” and click on “next step” and finish . Indexpattern created now, go in the left menu and click on “Discover” link to already view some data after selecting the index “log-messages”.

Open your browser and go on http://localhost:7500/v1/organization and you will see the application output:

{"result": "Wow! !:)"}

Go on Kibana and refresh data to see your logs data there. Hit again http://localhost:7500/v1/organization and go to view changes in Kibana

Find the source code of this article on Github

Thanks for reading. You can add comments if necessary to leave your appreciation. I will take your comments into consideration to improve this post.