Miscellaneous ways of Installation of Kafka on Ubuntu 18.04 for Novice Data Architect
By: Suraj Saha
Tags: Data Streaming, Big Data Streaming, Kafka Connect, Kafka, Kafka Installation

Photo by Lukas Blazek on Unsplash

I have kept this blog as short as possible on two commonly used ways of how to use Kafka Producer-Consumer processes over single node single broker acrhitecture with as minimal required features as possible to make it simple.

Prerequisite:

Ubuntu 18.04 server and a non-root user with sudo privileges. At least 4GB of RAM is required on the server. Installation without this amount of RAM may cause the Kafka service to fail, with the Java Virtual Machine(JVM) throwing out an “Out Of Memory” exception during startup. Even using Docker services one need to make sure the host machine has more than 4GB of RAM (advisable 8 GB RAM) as it is an absolute requirement for Kafka will consume a big part of RAM. OpenJDK 8/11 should be installed on the server. Kafka is written in Java, so it requires a JVM, however in order to use Confluent Open Source binaries one need to use Java 8.

Remember : While using sudo one must remember the root password.

1. Use Kafka setup files licensed by Apache

In order to install Apache Kafka as stated by official Kafka, we follow series of steps:

First Update the package repository cache of Ubuntu server with the following command :

$sudo apt-get update

Then we will install OpenJDK 8 or 11 on Ubuntu 18.04 server :

$ sudo apt-get intall openjdk-8-jdk

$ sudo apt-get install openjdk-11-jdk

Then we download the Kafka 2.4.0 (or any latest version as available) from Official website of Kafka :

$ sudo wget “http://apachemirror.wuchna.com/kafka/2.4.0/kafka_2.12–2.4.0.tgz” or $ sudo wget “http://mirrors.estointernet.in/apache/kafka/2.4.0/kafka_2.12_2.4.0.tgz”

After the installation we have to download the Zookeeper :

$ sudo wget “http://apachemirror.wuchna.com/zookeeper/stable/apache-zookeeper-3.5.6-bin.tar.gz” or $ sudo wget “http://mirrors.estointernet.in/apachezookeeper/”

Now we need to extract the kafka_2.12–2.4.0.tgz and apache-zookeeper-3.5.6-bin.tar.gz

$ sudo tar -xzf Downloads/kafka_2.12–2.4.0.tgz $ sudo tar -xzf Downloads/apache-zookeeper-3.5.6-bin.tar.gz

Now we enter into Kafka directory :

$ cd kafka_2.12–2.4.0.tgz

As Kafka uses Zookeeper so we need to first start a Zookeeper server, if we don’t have one. Always remember that Kafka broker runs over Zookeeper servers. It is Zookeeper server that is responsible for management of Kafka leaders and followers.

$ sudo bin/zookeper-server-start.sh config/zookeeper.properties

Now after Zookeeper instance is on, we use Kafka-server :

$ bin/kafka-server-start.sh config/server.properties

We create a topic named “Test123” :

$ sudo bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic Test123

We can check created topic by writing the command :

$ sudo bin/kafka-topics.sh --list --bootstrap-server localhost:9092

After running the producer we can write some messages:

$ sudo bin/kafka-console-producer.sh --broker-list localhost:9092 --topic Test123

and then we can consume the data by running a consumer process on another terminal.

$ sudo bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic Test123 --from-beginning

The demo of the above hand usage of Kafka for a single broker and node is shown below: