fabiochiu/medium-articles huggingface dataset:
Change Data Capturing with Debezium at willhaben
By: Maurizio Rinder
Tags: Jdbc, Postgres, Kafka, Data, Debezium

What we like about it…

With PostgreSQL 10 or higher, you already have all the basic requirements you need to get started.

A simple connector configuration is also quickly done and uploaded, and with that you can already see results in the Kafka topic.

Working on the database side, all the database changes are immediately written to the Kafka broker, making it quickly available for consumers. Consequently, for a microservice architecture, this is a neat way to transfer data to the service’s dedicated database with less effort. However that is a topic for a future blog post.

Debezium offers a way to split monoliths by easily transferring data changes to the new database while migrating

Since Debezium uses the write ahead log and replication slots, it also has a high degree of fault tolerance. Whenever a connector loses its connection to the database or the database crashes, the connector is always able to continue getting information from where it left off.

It is also worth mentioning that the documentation is properly written, and whenever you have problems or questions or need help, there is a dedicated Gitter support channel.

Although we experienced good times with Debezium, so far there are also been some downsides and areas that we think also need attention.

Problems we’ve encountered

In the beginning, while we were writing and uploading configuration files to Debezium, we obviously made some mistakes. However, the log messages often didn’t make clear what went wrong. So, there is a chance that a connector may crash and you will have to dig deep to find out what went wrong.

With that being said, at the beginning, our connectors crashed because we allowed the sending of tombstone records using the AvroConverter. The reason for that was because the connector wasn’t able to find a matching Avro schema.

Although Debezium is, in theory quite a failsafe tool, it also has a big downside concerning reliability. It is supposed to use a single database, which results in a single point of failure that is also not preventable with a database cluster, since you can’t easily hand over the job to a spare database.

Table-based messaging can be a big architectural concern for not using it

Last but not least there is the probability that you could expose your whole database structure to Kafka, which leads to bad encapsulation, since all consumers consume the database’s data fields in some way. Consequently, schema changes could become difficult to achieve after connecting many consumers to it. However, this problem can be prevented by using the Outbox pattern.
Что такое GraphQL и как это использовать в разработке приложений Apache Kafka
By: Nick Komissarenko
Tags: Kafka, Архитектура, Bigdata

Dell EMC Certified trainer since 2007, co-founder of Big Data School - delivering Big Data, Data Mining, Hadoop, NoSQL, Kafka, Spark cluster courses in Moscow!
Kafka handling large files
By: Narayana Basetty
Tags: Java, Kafka, Kinesis, Big Data, Streaming

Kafka can be tuned to handle large messages, so that kafka producers, brokers and consumers handled efficiently (A fine grained approach)

This can be done by configuring producer, broker, and consumer properties relating to maximum message and file sizes.

1. You try and reduce the size of messages first by applying compression formats.

2. Say if size of the reduced message still > 1MB then we need to build an algorithm to solve the problem.

3. Split your file into 1MB small chunks (1000000 bytes) [ex: 1000 x 1000]

Consider I have a json/xml file which is size : 1435KB

I can split this into 2 chunks[chunk 1, chunk 2]

Producer sends a chunk to the kafka broker on to the same partition(define a partition which I wish) and all these chunks which will have a message uuid, chunk type, chunk order , chunk size and aggregate at consumer side to form the original file data.

For example, if the original message is a text-based format (such as XML/JSON), in most cases the compressed message will be sufficiently small.

4. If producer published more than 1MB data then data loss occurs from kafka broker, can’t retrieve.
Improving your logs management solution
By: Ghaith Jemai
Tags: Beats, Kafka, Logstash, Elk, Kibana

Introduction:

In this post we will introduce one of the most popular logs centralization solution “ELK Stack”. Then we will improve it by adding new components and new features . Finally, we will give some tips in order to efficiently deploy your logs platform.

ELK :

As its name indicates, ELK solution is the acronym for the composition of the three components Elasticsearch-Logstash-Kibana tools.

ELK makes it easy for users to centralize your logs data, generated in heterogeneous format and coming from different sources, in one dashboard and offering the possibility of research, analysis and visualization of real time information system logs .

Elastisearch is an open source search and analysis engine written in java,

which uses Apache Lucene for data indexing . It stores information in a document-oriented format.

Logstash is an open source Extract-Transform-Load(ETL) tool written in Java and Ruby. Used for collecting, analyzing, managing, filtering and transforming logs.

Kibana is a data visualization tool , written in javascript. Used to visualize elasticsearch data.

It offers many features :

- Users can visualize the indexed data from elasticsearch in real time thanks to its powerful log exploration interface.

- Users can create bar charts, custom dashboards, etc …

- Users can create and apply filters on the data.

Why this solution is quite limited?

However, the ELK stack lacks some features and presents some drawbacks.

The most popular inconvenient of ELK is the huge memory consumption that logstash uses, users are always lead to set up logstash on each server that generates log files which causes a huge loss of resources for companies and may affect their applications.

In addition to that, this solution does not provide an identity and access management system in order to control users access to the company’s data.

Furthermore, this platform lacks from confidential data protection, it does not present an anonymization feature to hide clients sensitive information.

Finally, ELK does not provide an alerting real time system to detect and notify users in case of failures ,problems or weird spikes that could happen in production.

How to improve ELK?

Since this solution is not perfect, we focused on its limits and fixed them by building a new formula based on the ELK solution.

I want first to present the new Logs management architecture in the figure below:
Apache Kafka — Tutorial with Example
By: Andrea Scanzani
Tags: Software Development, Java, Kafka, Digital Architecture, System Architecture

Apache Kafka is an Event-Streaming Processing platform, designed to process large amounts of data in real time, enabling the creation of scalable systems with high throughput and low latency. Kafka provides the persistence of data on the cluster (main server and replicas) ensuring high-reliability.

In this post we will see how to create a Kafka cluster for the development environment, the creation of Topics, the logic of partitions and consumer groups, and obviously the publish-subscribe model.

The requirements to follow this tutorial are:

Java 8

Download latest Apache Kafka version

Note: All commands are launched on a Windows machine (KAFKA_HOME/bin/windows/), if you are on Linux use the path (KAFKA_HOME/bin).
Чем хорош REST Proxy для Apache Kafka и что с ним не так: ключевые достоинства и недостатки RESTful API от Confluent
By: Nick Komissarenko
Tags: Kafka, Архитектура, Bigdata

Dell EMC Certified trainer since 2007, co-founder of Big Data School - delivering Big Data, Data Mining, Hadoop, NoSQL, Kafka, Spark cluster courses in Moscow!
Streaming Pixel Server Data to Azure EventHubs via the Kafka Streaming Protocol | QueryClick
By: Phil Marius
Tags: Fluentd, Kafka, Azure Databricks, Azure, Spark Streaming

Implementation

So, what does this look like in practice?

Azure EventHubs Setup

First, I have to set up the AEH instance:

Create AEH instance

An AEH topic also needs to be created (yes, I know they’re not called topics, they’re called EventHubs. But that’s just confusing so I will refer to them as topics).

Create AEH topic

Fluentd Setup

Then, I have to configure Fluentd to utilise the out_kafka2 plugin. It comes built in with the td-agent installation of Fluentd, however, our pixel server uses Docker Compose with a Fluentd Docker image. Luckily, the official Docker image for Fluentd offers instructions for how to install plugins on their supplied image. This requires creating your own image so I followed the supplied instructions, added a gem install fluentd-plugin-kafka2 in it, and built the image locally (to be uploaded to Azure Container Repository once fully productionised).

This local image was then used in the docker-compose.yml in the pixel server:

services:

fkafka:

image: custom-fluentd:latest

restart: unless-stopped

volumes:

- ./fluentd/fluent.conf:/fluentd/etc/fluent.conf

- ./logs:/var/logs

env_file:

- .env

Then, in the mounted fluent.conf file, I added the new output plugin with AEH endpoint and credentials added under a <match> statement. I found this repo that helped.



@type copy

@type kafka2 # list of seed brokers

# port 9093 is used by Kafka

brokers <AEH_NAME>.servicebus.windows.net:9093

use_event_time true # buffer settings

buffer_type file

buffer_path /var/log/td-agent/buffer/kafka

flush_interval 3s # topic settings

default_topic <TOPIC_NAME>



</format> @type json # producer settings

max_send_retries 1

required_acks -1 # using default OS certs for SSL

ssl_ca_certs_from_system true username $ConnectionString

password "<SHARED_ACCESS_KEY>"

</store>



</store>

</match> @type stdout

A few credentials need to be obtained from AEH. Firstly, the broker connection string ( <AEH_NAME>.servicebus.windows.net:9093 ) is here:

AEH name credential location

Add in the topic name ( <TOPIC_NAME> ) from earlier. Then, create a shared access key ( <SHARED_ACCESS_KEY ) from the Shared Access Policies tab in AEH. Make sure it has Send permissions:

Create shared access key

Databricks Setup

Now for Databricks, I followed this tutorial and ended up with a setup like this:

Databricks streaming setup

You’ll need to create another shared access key that can Listen from the previous step for this. Make sure you specify the EntityHub to be the topic name in the connection string. Also, new versions of Spark require the connection string to be encrypted, use sc._jvm.org.apache.spark.eventhubs.EventHubs.Utils.encrypt from the Spark Context sc to achieve that:

conn_string = "<SHARED_ACCESS_KEY>" conf = {

"eventhubs.connectionString": sc._jvm.org.apache.spark.eventhubs.EventHubsUtils.encrypt(conn_string)

}

(It’s honestly this simple)
Что такое REST Proxy к Apache Kafka: разбираемся с RESTful API от Confluent
By: Nick Komissarenko
Tags: Kafka, Архитектура, Bigdata

Dell EMC Certified trainer since 2007, co-founder of Big Data School - delivering Big Data, Data Mining, Hadoop, NoSQL, Kafka, Spark cluster courses in Moscow!
整合：Spring Boot AWS MSK. 2020年底了，最近做了一次 Spring Boot 與 AWS MSK…
Tags: AWS, Spring Boot, Kafka

Learn more. Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Make Medium yours. Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore
Как удаленному сервису достучаться к Apache Kafka: HTTP или REST API
By: Nick Komissarenko
Tags: Kafka, Python, Bigdata

Dell EMC Certified trainer since 2007, co-founder of Big Data School - delivering Big Data, Data Mining, Hadoop, NoSQL, Kafka, Spark cluster courses in Moscow!
Connecting Kafka to Oracle Data Catalog
By: Eloi Lopes
Tags: Oracle Cloud, Kafka, Data Catalog, Oracle, Big Data

Today, I going to show how to connect Kafka to Oracle Data Catalog. Kafka is running on Oracle Big Data Service.

What is Oracle Data Catalog?

Oracle Cloud Infrastructure (OCI) Data Catalog is a metadata management service that helps data professionals discover data and support data governance.

To connect to Data Catalog, I’m using the public IPs of Big Data Cluster. To configure these IPs you need to create it manually. Check the documentation in case you haven’t created.

Configuring Kafka

Go to your Cloudera Manager and click on Kafka:

Go to instances and you will have two or more Kafka brokers. You need to add this configuration to each broker instance:
Spring Cloud Stream Kafka — Atypical Configuration Strategy
By: Vinodh Subramanian
Tags: Cloud Configuration, Scdf, Kafka, Spring, Spring Cloud Stream

We’ll be discussing only the way how to configure the binders and bindings using a configuration class. And this blog is completely based on Kafka as the binder.

CloudStreamConfiguration

I have got this configuration class SpringCloudStreamConfig.java which holds all the destination binders and bindings properties defined at one place. Let’s get started with configuring bindings.

Destination Bindings: Bridge between the external messaging systems and application provided Producers and Consumers of messages (created by the Destination Binders).

Binding Configuration

We have to define a Bean of type BindingServiceProperties. Since there is already a spring defined bean, we should make sure we have made our custom defined BindingServiceProperties Bean as Primary. For this article, we’ll use two topics test1 and test2 as the bindings. A typical binding will be requiring destination — topic name, group — consumer-group name, consumer — all the consumer-related properties, and the binder — name of the binder which we are binding in our application. Since we are using a single binder, we can ignore the binder property. It’s pretty much straight forward implementation here. Let's move on to configuring our binder now.

Destination Binders: Components responsible to provide integration with the external messaging systems.

Binder Configuration

Similar to BindingConfiguration we have to create a Bean of type KafkaBinderConfigurationProperties and annotate with @Primary to override the default spring implementation. There are a lot of configuration associated with Kafka Binders, but for our example, we can just stick with the two most important properties — brokers and defaultBrokerPort. And that's the end of it. Boot up your application with this configuration class in place and everything should fall in place.

That’s it for this post. The complete example can be found here. Feel free to comment on the content.
Backup System for Apache Kafka ( Part 1 )
By: Davinder Pal
Tags: Python, S3, Kafka, AWS, Backup

Honestly, there aren’t many good backup tools for Apache Kafka but why we need a backup tool in first place. As Apache Kafka didn’t had any offsite backup solution or DR solutions for it, there are tricks which can be done but none are 100% reliable.

Example Trick:

create MM1 cluster and it will sync data one-way to another Apache Kafka Cluster Only.
It Is Often Safer To Be In Chains Than To Be Free
By: Laressa Alkmim
Tags: Culture Change, Kafka, Behavior Change, Feminism, Writing

This is the why of the polarization of the world, some people want to be free, others want to be safe.

Photo by Saketh Garuda on Unsplash

Hi,

To explain better the title of this article its a phrase of Franz Kafka from his book THE TRIAL. But this phrase serves to ilustrate quite good the right and left thinking in the actual world.

The chains represent everything that is a prision, but a the same time protect us from the dangers of the open world. We can’t deny that the world is a dangerous place, with his field full of traps and diferent things that we don’t understand.

A good example of this fear is a teenager that prefers to be at his parents house and follow the rules than be thrown away to fend for himself.

But some of the teenagers don’t have choice, some survive by themselves and thrive, others just sink to a deep hole of desperation.

Photo by Shane on Unsplash

Another example is the feminism and the anti-feminism, if you talk to a feminist they will say: what the hell a woman is thinking to be an anti-feminist?

Photo by Jonathan Borba on Unsplash

Well I had the curiosity to read a paper that an anti-feminist wrote about why she is anti-feminist, in this paper she said that she wants someone to protect her, to work and bring what she needs to home and then she will be safe just taking care of children and housewifer.

She wants to be protected and cherished. Thats not crazy. It is hard to have to work and take care of children and housewifer!

But what this woman do not count on is that life is full of surprises and be at the mercy of someone is not very good.

In the case of feminism, be at mercy of men in general is not very fun, when you want something more than to be at home and have children. What happens if you don’t want to be a housewife?

Photo by Denys Nevozhai on Unsplash

Lets be honest, every human being whants to be at the center of his universe, so if you are under someone’s comand is better to be of use.

Simplifying if you are under someone’s comand you are under their protection, but you have to be of use and follow the rules.

On the other hand if you chose to be free, you have no protection, but you have to be of use for yourself and follow your on rules. Things can go South, but they will go South on your on way.
Great content.
By: Prakash P
Tags: Kafka, Technical Writing

Great content. We need more articles from authors like you. Thank you for all your contributions so far.
DataflowとMSKを用いたログ収集システムのリプレース
By: Yuuki Sasaki
Tags: Terraform, Dataflow, Bigquery, Kafka, Apache Beam

Learn more. Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Make Medium yours. Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore
От open-source до Confluent: 3 клиента Python для Apache Kafka
By: Nick Komissarenko
Tags: Kafka, DevOps, Bigdata

Dell EMC Certified trainer since 2007, co-founder of Big Data School - delivering Big Data, Data Mining, Hadoop, NoSQL, Kafka, Spark cluster courses in Moscow!
Как опередить спрос на модные новинки с облачными технологиями Big Data: кейс компании Boden по Apache Kafka и Snowflake
By: Nick Komissarenko
Tags: Kafka, Dwh, Bigdata

Dell EMC Certified trainer since 2007, co-founder of Big Data School - delivering Big Data, Data Mining, Hadoop, NoSQL, Kafka, Spark cluster courses in Moscow!
Data Migration from Postgres using Kafka Connect
By: Zeeshan Kareem
Tags: Kafka Connect, Postgres, Kafka

Overview: This story covers the process of automatic data migration from one database (here Postgres) to another using Kafka Connect for Change Data Capture (CDC). The source consists of concerned list of tables by creating Publication in Postgres source Database1 whose data needs to be migrated into certain tables of Database2.

Tools: Postgres, Kafka Connect, Kafka, Spring Boot, Java

Diagram representation of data flow

Step 1: Create Publication for the concerned list of tables in Database

CREATE PUBLICATION dbz_publication FOR TABLE table1, table2, table3,…., tableN;

Note: Until and unless PUBLICATION name is explicitly mentioned while setting Kafka Connect configuration, it should be dbz_publication in order to caputure changes in Database1 by Kafka Connector.

Step 2: Connect the Postgres Database1 as a source to Kafka

Use Postman to configure settings needed to sink data from required tables of Database1 to Kafka:

POST request: http://×××:×××/connectors

BODY (raw):

{“name”: “postgres-source”,

“config”: {“connector.class”:”io.debezium.connector.postgresql.PostgresConnector”,

“tasks.max”:”1",

“database.hostname”: “×××”,

“database.port”: “×××”,

“database.user”: “×××”,

“database.password”: “×××”,

“database.dbname” : “Database1”,

“database.server.name”: “dbserver1”,

“database.whitelist”: “Database1”,

“database.history.kafka.bootstrap.servers”: “×××:×××”,

“database.history.kafka.topic”: “schema-changes.Database1”,

“key.converter”: “org.apache.kafka.connect.json.JsonConverter”,

“value.converter”: “org.apache.kafka.connect.json.JsonConverter”,

“key.converter.schemas.enable”: “false”,

“value.converter.schemas.enable”: “true”,

“value.converter.schema.registry.url”: “http://×××:8081”,

“plugin.name”: “pgoutput",

“publication.autocreate.mode”: “filtered”

}

}

Note: We are using standard logical decoding plugin pgoutput which transforms the changes read to the logical replication protocol and filters the data according to the publication specification. To check if the above registered settings have been configured, hit a GET request from Postman with URL http://×××:×××/connectors/postgres-source/status

Step 3: Check the data streaming in topic(s)

Use Kafka Tool 2 to verify the data from the topics generated by published tables

Note: In order to read data properly, Key and Value should be converted to String from Kafka Tool 2 settings.

Step 4: Processing of streaming data from Kafka

Use consumer service(s) for writing business logic in order to read data from the topic(s), then process it as per the requirement and persist it in Database2.

Note: In my case, I had a Spring Boot microservice with consumers written in Java. While consuming data from a certain topic, three cases can be made for separate processing, i.e., Add, Update and Delete and then persisted in respective tables of Database2.

Step 5: Verify Data Migration from Database1 to Database2

Write DML statement(s) in Database1 and check if the corresponding changes (as per business logic in Step 4) are reflected in Database2
MYSQL Kafka Connect Tutorial on Docker
Tags: Mysql 8, Kafka Connect, Kafka

In this tutorial, we will use docker-compose, mysql 8 as examples to demonstrate Kafka Connector.

This tutorial is mainly based on the tutorial written on https://docs.confluent.io/5.0.0/installation/docker/docs/installation/connect-avro-jdbc.html. However, the original tutorial is out-dated that it just won’t work if you followed it step by step. This is a refreshment of that tutorial, also to simplify things, we will get rid of the Avro set up as it serves no purpose to demonstrate Kafka Connector. Enough said, let’s begins.
Go-Kafka-Go
By: Karan Churi
Tags: Go, Golang, Docker Compose, Kafka, Advertised Listeners

Let's get started with server setup via docker.

Here we’ll need to spin up following container.

Zookeeper

Kafka

Below is a docker-compose file to get both running.

In the above docker-compose file, the environment of each container plays an important role. I’ve struggled for some hours to get it working, I’ll explain some of its properties.

ZOOKEEPER_SERVER_ID identifies each zookeper differently.

KAFKA_BROKER_ID identifies each kafka broker differently.

ZOOKEEPER_SERVERS links of all zookeeper, so that they can communicate within themselves.

KAFKA_ZOOKEEPER_CONNECT links of all zookeeper

Here the link contains the internal address of zookeper within the docker.

For eg. zk1:22888

Here zk1 corresponds to docker container with name as zk1

Now the tricky part, Listener:

We’ve configured 2 types of listener.

Internal [Within docker environment]

External [Outside docker environment …. i.e via localhost]

We need to map our custom listener with the kafka’s security protocol either internally or externally. We will be using PLAINTEXT security protocol in this example

In our example LISTENER_INTERNAL and LISTENER_EXTERNAL are custom protocols, we need to map it to PLAINTEXT. To do so, below line is used

KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_INTERNAL:PLAINTEXT,LISTENER_EXTERNAL:PLAINTEXT

Now items in KAFKA_LISTENERS are mapped with KAFKA_ADVERTISED_LISTENERS sequentially.

So LISTENER_INTERNAL://kafka-1:19092 will be mapped with LISTENER_INTERNAL://kafka-1:19092

And LISTENER_EXTERNAL://kafka-1:19091 will be mapped with LISTENER_EXTERNAL://localhost:19091

Pay attention to the change in localhost part.

Now that we are done with mapping listener, we need to specify kafka which among 2 listeners to use for internal communication. To do so, below line is used.

KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_INTERNAL

Well than, that's it with our server setup.
Kafka
By: Pramod Narayana
Tags: Producer, Consumer, Introduction To Kafka, Kafka

This is a series of articles on Kafka. Lets start will a brief introduction to Kafka

Introduction to Kafka

Kafka is a distributed circular persistent message queue. A queue is divided and distributed across the nodes in the cluster. Old messages are deleted after a certain time to make room for new messages. Hence it is called a circular message queue. By default, old messages are deleted after 1 week. All the messages are stored on the disk. Hence it is a persistent message queue. Also for faster access, messages are available in memory

Producer Application Examples

Credit card application will send credit card swipes as messages to Kafka Ad-server Application will send ad impressions and ad clicks as messages to Kafka. Sensor Application will send sensor data as messages to Kafka

Consumer Application Examples

Spark Streaming, Flink Streaming, Samza, Kafka Streaming can consume messages from Kafka, do ETL transformation, and save transformed messages back to Kafka or other external storage in real-time. Web frameworks can also consume messages from Kafka and display them on websites in real-time.

Github repository

https://github.com/pixipanda/kafkatraining

https://github.com/pixipanda/avro-consumer-app

https://github.com/pixipanda/avro-consumer-app2

Ubuntu image

Download ubuntu image (3.86GB)

username: hduser

password: hadoop123

IntelliJ

IntelliJ is installed in the home directory (/home/hduser/idea-IC-183.6156.11/bin)

Go to IntelliJ’s bin directory

cd idea-IC-183.6156.11/bin/

Start IntelliJ

./idea.sh

First, create a workspace directory in your home directory

mdkir workspace

cd workspace

Clone kafkatraining repository from GitHub

git clone https://github.com/pixipanda/kafkatraining

Import Code to IntelliJ

Open IntelliJ and Click on Import Project

Select the downloaded Project and Click OK

Checkmark “Import Project From External Model”

Select Maven and Click OK

Checkmark “Search for projects recursively” and

Checkmark “Import Maven projects automatically” and Click Next

By default, your project will be selected. Click Next

Add New JDK by Clicking on the “+” mark on the top left corner

By default, the java installation directory will be selected. Click Next.

JDK 1.8 will be loaded. Click Next

Let the default Project name and Project Location be as it is and Click Next

Finally, all the dependency libraries will be downloaded and the project will be loaded

Let’s see the components of Kafka in the next article

Summary
Kafka Streams with Java 15
By: Diego Pacheco
Tags: Java, Kafka, Kafka Streams, Java15, Streaming

Kafka Streams is a Streaming library similar to Spark and Flink which works with Apache Kafka. Kafka stream can be useful to process historical and near-real-time big data workloads but also for non-realtime analytical computations at scale for the online world as well. Kafka-Streams is elastic, highly scalable, fault-tolerant, and fully integrated with Kafka. You can use Kafka Stream with Java, Scala, Kotlin JVM applications and also have exactly-once processing semantics. Kafka-Streams is being used by the New York Times, Pinterest, Line, Trivago, Zalando, and many other companies. Today I want to share a video of Kafka-Streams running with Kafka 2.6 and Java JDK 15. So Let’s get started.

The Video

The Code

Cheers,

Diego Pacheco
How to run ZooKeeper and Kafka in Docker
By: Natthapon Pinyo
Tags: Docker, Localhost, Zookeeper, Kafka

There are so many Docker images for Kafka and ZooKeeper.

One of the most popular Docker Image is Bitnami/kafka

From an official document, just few steps to do it.

Basically we need to create Docker Network to be a bridged between ZooKeeper and Kafka Docker Containers.

Step 1 — Create Docker Network

$ docker network create app-tier --driver bridge

This step is to create Docker Network named app-tier

Step 2 — Launch ZooKeeper container

$ docker run -d --name zookeeper --network app-tier -p 2181:2181 -e ALLOW_ANONYMOUS_LOGIN=yes bitnami/zookeeper

This step is to create Docker Container from bitnami/zookeeper inside Docker Network app-tier with port mapping 2181 to localhost 2181

Step 3 — Launch Kafka container

$ docker run -d --name kafka --network app-tier --hostname localhost -p 9092:9092 -e ALLOW_PLAINTEXT_LISTENER=yes -e KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181 bitnami/kafka

This step is to create Docker Container from bitnami/kafka inside Docker Network app-tier with port mapping 9092 to localhost 9092 and connect to zookeeper container in the same Docker Network

You can check running Docker Instance from Docker Dashboard

Docker Dashboard

To connect Kafka, you can point you application config to localhost:9092

That’s it! Happy coding :)
Как устроен конвейер аналитики больших данных на Apache Kafka и Druid в Netflix
By: Nick Komissarenko
Tags: Druid, Kafka, Bigdata

Dell EMC Certified trainer since 2007, co-founder of Big Data School - delivering Big Data, Data Mining, Hadoop, NoSQL, Kafka, Spark cluster courses in Moscow!
Streaming Data Analyze (Anlık Veri Analizi)
By: Gürkan Güney
Tags: Data Engineering, Spark, Kafka, Big Data, Streaming

In this project that I have realized, it is inspired by a scenario that can be in daily life and the data is analyzed instantly.

The project was made on HDP (Hortonworks Data Platform) 2.6.5. Generally, I used tool like Hadoop file system, Spark, Kafka, Hive, Zeppelin and Nifi.

In project, We enter manually data to Kafka topic, we get data from Nifi with Nifi’s consume kafka feature, we parse data with Nifi’s route on content feature, we send data to different two Kafka topic, we save the data on these topics under a file in the Hadoop file system, we listen instantly that file with Spark Streaming and save else folder.
Install Kafka Cluster and Zookeeper with High Availability
Tags: High Availability, Zookeeper, Kafka

kafka multi broker

What is apache kafka ?

Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. https://kafka.apache.org/

What is apache zookeeper?

ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications. https://zookeeper.apache.org/

Setup Zookeeper Cluster with High Availability

Install java on all node ha-zoo*

sudo apt update

apt install -y openjdk-11-jdk

setup dns local on all node ha-zoo*

vi /etc/hosts

10.20.20.51 ha-zoo1

10.20.20.52 ha-zoo2

10.20.20.53 ha-zoo3

download zookeeper on all node ha-zoo*



sudo wget cd /optsudo wget https://downloads.apache.org/zookeeper/zookeeper-3.6.2/apache-zookeeper-3.6.2-bin.tar.gz

ekstract zookeeper package on all node ha-zoo*

sudo tar -xvf apache-zookeeper-3.6.2-bin.tar.gz

sudo mv apache-zookeeper-3.6.2 zookeeper

cd zookeeper

config multiple zookeeper on all node ha-zoo*

sudo vi conf/zoo.cfg

tickTime=2000

dataDir=/data/zookeeper

clientPort=2181

initLimit=10

syncLimit=5

server.1=ha-zoo1:2888:3888

server.2=ha-zoo2:2888:3888

server.3=ha-zoo3:2888:3888

set multiple zookeeper, setup each node

on node ha-zoo1, add the file

vi /data/zookeeper/myid

1

on node ha-zoo2, add the file and save

vi /data/zookeeper/myid

2

on node ha-zoo3 add the file and save

vi /data/zookeeper/myid

3

running zookeeper on all node ha-zoo*

java -cp lib/zookeeper-3.6.2.jar:lib/*:conf org.apache.zookeeper.server.quorum.QuorumPeerMain conf/zoo.cfg

testing on node ha-zoo3

cd /opt/zookeeper

bin/zkCli.sh -server ha-zoo1:2181

[zk: ha-zoo3:2181(CONNECTED) 0] ls /

[zookeeper]

[zk: ha-zoo3:2181(CLOSED) 3] quit

well done, zookeeper cluster running well. And then we will setup kafka cluster.
Spring Boot app with Apache Kafka in Docker container
By: Artiom Mozgovoy
Tags: Docker, Spring Boot, Docker Compose, Java, Kafka

Spring Boot Java client

Let’s write a minimalistic client to publish/consume messages from kafka. To do this, we will create a new project through Spring Initializr or any other tool you are used to. It will be enough to connect one dependency:

spring-kafka

I prefer to configure Spring Boot applications via yaml config files with minimal Java configurations. Due to the built-in structuring in Yaml-format, these files are much more human-readable than .properties files and allow you to group settings by modules. The only drawback of the .yml configuration is that it is not supported in tests, but it is easily bypassed by the built-in Spring Boot tools.

I highly recommend to try and check how your perception of the old .properties file changes (especially for large projects) if you rewrite it in yaml. This can be done in a couple of clicks at https://www.toyaml.com/index.html.

In project resources create an application.yml file with the following content:

This setup works for sequentially reading one message at a time (max-poll-records: 1) by one listener (concurrency: 1) and requires manual confirmation of message processing in listener (ack-mode: manual_immediate).
Building a Simple Kafka Client for the Web and Desktop
By: Dânesh Hussain Zaki
Tags: Kafka, JavaScript, Tutorial, Web Development

I wrote a short tutorial on building a simple, native Kafka client using Web Sockets and ElectronJS, a few weeks ago. Please read it if you are interested.

https://dzone.com/articles/building-a-simple-kafka-client-for-the-web-and-des
Apache Kafka Introduzione Tutorial e Primi Passi
By: Andrea Scanzani
Tags: Software, Architecture, Java, Kafka, Digital Architecture

Learn more. Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Make Medium yours. Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore
Зачем вам UNION вместо JOIN в Apache Druid и семплирование больших данных в Spark Streaming: пример потоковой аналитики Big Data
By: Nick Komissarenko
Tags: Druid, Kafka, Bigdata

Dell EMC Certified trainer since 2007, co-founder of Big Data School - delivering Big Data, Data Mining, Hadoop, NoSQL, Kafka, Spark cluster courses in Moscow!
Why Kafka ? What does it solve?
By: Abdennour Toumi
Tags: Site Reliability, Ask Why, Kafka

Each system can produce data, consume data or both.

Integrating between systems becomes a must.

If system A produces data, and System B & C needs this data, you will have to integrate A +B , then A+C.

Integrating systems costs a lot with this way ( i mean two by two).

- Imagine you have 4 tools, the number of possible integration between every two is : C(4,2) = 6

- Imagine you have 10 tools, the number of possible integration between every two is : C(10,2) = 45

Note: C(n,k) is a maths formula, check it here: https://www.calculatorsoup.com/calculators/discretemathematics/combinations.php

Kafka comes to avoid all these integration. Instead of integrating two by two, you just need to put Kafka as centralized place for Data Streaming : Data producers ( systems which produces data) need to report their data to Kafka & Data consumers ( systems which needs this data) they just need to subscribe to Kafka ( e.g Kafka topics,… so on)
Почему вам нужно обучение разработке в Apache Kafka: 4 причины выбрать курсы от Школы Больших Данных
By: Nick Komissarenko
Tags: Большиеданные, Kafka, Bigdata

Dell EMC Certified trainer since 2007, co-founder of Big Data School - delivering Big Data, Data Mining, Hadoop, NoSQL, Kafka, Spark cluster courses in Moscow!
Kafka: Steps to increase Replication Factor for a topic
By: Varun Pal
Tags: Replication Factor, Kafka

Below are the steps to increase the Replication Factor for a topic

To list all the topics in your cluster

$KAFKA/kafka-topics.sh — zookeeper $ZOOKEEPER — list

To view all the default and custom configurations for a topic

$KAFKA/kafka-topics.sh — zookeeper $ZOOKEEPER — topic <topic_name> — describe

The first line of the output describes the configs. For example:

Topic: <topic_name> PartitionCount: 8 ReplicationFactor: 2 Configs:

The above output tells us that this topic has a Partition Count as 8 with a Replication Factor of 2.

Say we would like to increase the Replication Factor to 3 for this topic. In which case, we need to construct the following increase-replication-factor.json file that elaborates the increase in Replication Factor from 2 to 3.

cat increase-replication-factor.json

{“version”:1,

“partitions”:[

{“topic”:”<topic_name>”,”partition”:0,”replicas”:[1,2,3]},

{“topic”:”<topic_name>”,”partition”:1,”replicas”:[1,2,3]},

{“topic”:”<topic_name>”,”partition”:2,”replicas”:[1,2,3]},

{“topic”:”<topic_name>”,”partition”:3,”replicas”:[1,2,3]},

{“topic”:”<topic_name>”,”partition”:4,”replicas”:[1,2,3]},

{“topic”:”<topic_name>”,”partition”:5,”replicas”:[1,2,3]},

{“topic”:”<topic_name>”,”partition”:6,”replicas”:[1,2,3]},

{“topic”:”<topic_name>”,”partition”:7,”replicas”:[1,2,3]}

]}

To execute the replication execute the following command:

$KAFKA/kafka-reassign-partitions.sh — zookeeper $ZOOKEEPER — reassignment-json-file $CONFIG/increase-replication-factor.json — execute

To view the progress of the process the following command can be used

$KAFKA/kafka-reassign-partitions.sh — list — bootstrap-server $BROKER — command-config $CONFIG/client.properties

You can verify the progress of the process via the following command

$KAFKA/kafka-reassign-partitions.sh — zookeeper $ZOOKEEPER — reassignment-json-file $CONFIG/increase-replication-factor.json — verify

Once the replication factor is increased the above command will display the following output

Status of partition reassignment:

Reassignment of partition <topic_name>-0 is complete.

Reassignment of partition <topic_name>-1 is complete.

Reassignment of partition <topic_name>-2 is complete.

Reassignment of partition <topic_name>-3 is complete.

Reassignment of partition <topic_name>-4 is complete.

Reassignment of partition <topic_name>-5 is complete.

Reassignment of partition <topic_name>-6 is complete.

Reassignment of partition <topic_name>-7 is complete.

The following environment variables have been considered

$KAFKA — location of the bin directory /root/kafka/bin

$CONFIG — location of config directory /root/kafka/config

$ZOOKEEPER — zookeeper string in the localhost:port format

$BROKER — broker string in the localhost:port format
Apache Kafka Fundementals Part-1
By: Yamen Şaban
Tags: Analytics, Apache Kafka, Kafka, Big Data, Data Science

Understanding the Basic Principles of Messaging Systems:

You would not have come here if you did not know about applications where each application uses data that gets processed by other external applications or application consuming data from one or more external data sources.

Messaging System acts as an integration channel for data exchange between such applications.

In application integration systems design, there are a few important principles that should be kept in mind:

• Loose coupling: The implement of this concept ensure minimal dependencies of application between each other. Tightly coupled applications are coded based on predefined specifications of other applications, any change in the specifications will result in a break in the chain between the application.

• Common interface definition: Ensure a common agreed-upon data format for exchange between applications, this helps to establish message exchange standards among applications, ensures that some of the best practices of information exchange can be enforced easily.

Apache Avro is a good choice for message exchanges as it serialize data in compact binary format and support schema evaluation.

• Latency: The time taken by messages to be transferred between the sender and the receiver high latency is not a desirable thing even in an asynchronous mode of communication.

• Reliability: Is making sure that the temporary unavailability of applications does not affect other dependent applications that need to exchange data.
Brief dive into message queues (pub-sub)
By: Nicolas Wee
Tags: Pub Sub, At Least Once, Message Queue, Kafka, Exactlyonce

Message queues or some would call it, publish-subscribe pattern (pub-sub) is an asynchronous messaging patterns that is often used to send information between microservices. We will go through how pub-sub will work and the advantages and disadvantages of using a pub-sub pattern.

How does message queues work?

Fig 1 Simple Message Queue

Message queues like Kafka or RabbitMQ acts as the middle man when one microservice send a message to another microservice. This allows asynchronous calls to be made between microservices hence improving performance. There are a 3 main Quality of Service, At most once, At least once, Exactly once.

At most once, is the lowest-overhead required where the client simply just publishes the message and there is no acknowledgement by the broker.

At least once, guarantees that the message is send successfully to the broker as the broker acknowledge back to the sender. If the sender do not receive the acknowledgement by the broker, it will send the message again.

Exactly once, is the highest level of service where a handshake is needed between the sender and broker to confirm that the main message has been sent and received. If the handshake is not completed, the message will be send again.

Advantages

Loose Coupling

Responsibilities between the sender and subscriber are separated as the sender do not require information about the subscriber. They both have the capabilities to perform their responsibilities without the need of the other.

Performance

Asynchronous calls will allow the microservice a lower latency from receiving the call to sending a response. Complex jobs which do not require an immediate response can be segregated into another microservices.

Disadvantages

Reliability

Reliability of messages received by the subscriber depends on the performance scarified. If exactly once principle is used, the efficiency of the asynchronous would not be as distinct.

Security

There is a possibility where a middle-man attack might intruder and obtain the messages being send out to the subscribers. A hacker can even spoof himself as the publisher and send malicious message into the message queue.
How to authenticate Kafka using Kerberos (SASL) with terminal, Spark (and Spark Streaming) and Jupyter Notebook.
By: Artem Gogin
Tags: Spark, Kafka, Kerberos, Big Data, Spark Streaming

Let’s do the same thing using Spark.

The challenge here is that we want Spark to access Kafka not only with application driver, but also with every executor. It means, each executor needs to obtain ticket from Kerberos with our keytab. To make Spark do this, we need to specify right configurations.

Firstly, we need the same jaas.conf:

KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab=”${YOUR_KEYTAB_FILE} “ principal=”${USER_NAME}@${REALM}”; };

Before launching Spark, we also need to export the variable:

export SPARK_KAFKA_VERSION=0.10

In Spark code we will access Kafka with these options (the first 5 is mandatory):

kafka.bootstrap.servers=${KAFKA_BROKERS_WITH_PORTS} kafka.security.protocol=SASL_PLAINTEXT kafka.sasl.kerberos.service.name=kafka kafka.sasl.mechanism=GSSAPI subscribe=${TOPIC_NAME} startingOffsets=latest maxOffsetsPerTrigger=1000

You can pass these options map to:

spark.readStream.

format("kafka").

options(myOptionsMap).

load()

Before starting Spark we can define shell variable

JAVA_OPTIONS="-Djava.security.auth.login.config=jaas.conf -Djava.security.krb5.conf=/etc/krb5.conf"

Also we will need two copies of users Keytab with different name. If we already have one, we can create the second one with command:

cp $USER_NAME.keytab ${USER_NAME}_2.keytab

And to launch the spark application we should run this command:

spark2-submit \ --master yarn \ --conf "spark.yarn.keytab=${USER_NAME}_2.keytab" \ --conf "spark.yarn.principal=$USER_NAME@$REALM" \ --conf "spark.driver.extraJavaOptions=$JAVA_OPTIONS" \ --conf "spark.executor.extraJavaOptions=$JAVA_OPTIONS" \ --class "org.example.MyClass" \ --jars spark-sql-kafka-0-10_2.11-2.4.0.jar \ --files "jaas.conf","${USER_NAME}.keytab" \ my_spark.jar

Or you can use the same configurations with spark-shell or pyspark.

Note: to allow Spark access HDFS we specify spark.yarn.keytab and spark.yarn.principal. To allow Spark access Kafka we specify spark.driver.extraJavaOptions and spark.executor.extraJavaOptions and provide files jaas.conf, ${USER_NAME}.keytab, mentioned in JavaOptions so every executor could receive a copy of these files for authentication. And for spark kafka dependency we provide spark-sql-kafka jar suitable for our spark version. We can also use option — package instead of — jars.

Hope everything worked!
Веб-реклама, ретаргетинг и проблемы потоковой аналитики больших данных с Apache Kafka, Spark Streaming и Druid: кейс платформы Outbrain
By: Nick Komissarenko
Tags: Druid, Kafka, Bigdata

Dell EMC Certified trainer since 2007, co-founder of Big Data School - delivering Big Data, Data Mining, Hadoop, NoSQL, Kafka, Spark cluster courses in Moscow!
Fixing “Classpath is empty. Please build the project first e.g. by running ‘gradlew jarAll’ ” and “The System cannot find the path specified” errors on Windows Confluent Kafka
By: Jabiulla Shaik - Jb
Tags: Classpath, Windows, Confluent Kafka, Apache Kafka, Kafka

Photo by James Lee on Unsplash

Confluent Kafka provides a set of bat files to work on Windows platform. When you try to run these bat files, If everything works fine, then it’s good. Sometimes you run into below errors.

Classpath is empty. Please build the project first e.g. by running ‘gradlew jarAll’ The System cannot find the path specified

Error 1: Classpath is empty. Please build the project first e.g. by running ‘gradlew jarAll’

Classpath is empty error

Now Let’s see, how to fix this error.

Open kafka-run-class.bat file. You will find this in bin\windows folder of your confluent Kafka.

Note : Under bin folder you will find all the shell scripts that can be used on Linux platform and under bin\windows you will find the bat files that you can use on Windows platform.

Search for “rem Classpath addition for core” in kafka-run-class.bat file. When you find it, it looks like below.

Now lets add the below snippet above the “rem Classpath addition for core” line in kafka-run-class.bat file.

rem classpath addition for LSB style path

if exist %BASE_DIR%\share\java\kafka\* (

call:concat %BASE_DIR%\share\java\kafka\*

)

After adding the above snippet, your file will look like this.

All kafka jar files are located inside share\java\kafka folder. We are just adding this classpath so that the classpath error (Classpath is empty. Please build the project first e.g. by running ‘gradlew jarAll’) goes off.

Now, when you run the bat files under windows folder, they will run fine without any issues. Here is the zookeeper start log after fixing the classpath error.