Install Apache Kafka on Windows 10 - onlinetutorialspoint

PROGRAMMINGJava ExamplesC Examples
Java Examples
C Examples
C Tutorials
aws
JAVAEXCEPTIONSCOLLECTIONSSWINGJDBC
EXCEPTIONS
COLLECTIONS
SWING
JDBC
JAVA 8
SPRING
SPRING BOOT
HIBERNATE
PYTHON
PHP
JQUERY
PROGRAMMINGJava ExamplesC Examples
Java Examples
C Examples
C Tutorials
aws
This guide helps you to understand how to install Apache Kafka on Windows 10 operating system and executing some of the basic commands on Kafka console.
Apache Kafka is a distributed, fast and scalable messaging queue platform, which is capable of publishing and subscribing to streams of records, similar to a message queue or enterprise messaging system.
Install JRE before you further.
Download the latest Apache Kafka from the official Apache website for me it is 2.11.2.0.0 release.
 
Click on above highlighted binary downloads and it will be redirected to Apache Foundations main downloads page like below.
Select the above-mentioned apache mirror to download Kafka, it will be downloaded as a .tgz. Extract it and you will see the below folder structure.
/bin directory represents all the binary files which are helpful to start Kafka server different operating systems. As we are working with the windows machine, there will be a folder named windows under /bin directory, which has all the windows related stuff.
/config directory contains all configuration details about Kafka server, zookeeper, and logs. All configurations have their default values if you wanted to change any config details like port you can freely go and change accordingly.
# The directory where the snapshot is stored.
dataDir=/tmp/zookeeper
# the port at which the clients will connect
clientPort=2181
# disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=0
server.properties
############################# Zookeeper #############################

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=localhost:2181

# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000

############################# Log Basics #############################

# A comma separated list of directories under which to store log files
log.dirs=/tmp/kafka-logs
/libs directory contain all dependency executables like java, jetty, log4j and etc.
If you are okay with the above default configurations, you are ready to start zookeeper and Kafka server.
Go to Kafka_X.XX-X.X.X\bin\windows\
D:\Softwares\kafka_2.11-2.0.0\bin\windows>zookeeper-server-start.bat ../../config/zookeeper.properties
[2018-11-17 03:20:58,713] INFO Reading configuration from: ..\..\config\zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2018-11-17 03:20:58,713] INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.DatadirCleanupManager)
[2018-11-17 03:20:58,713] INFO autopurge.purgeInterval set to 0 (org.apache.zookeeper.server.DatadirCleanupManager)
[2018-11-17 03:20:58,713] INFO Purge task is not scheduled. (org.apache.zookeeper.server.DatadirCleanupManager)
[2018-11-17 03:20:58,713] WARN Either no config or no quorum defined in config, running  in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
[2018-11-17 03:20:58,744] INFO Reading configuration from: ..\..\config\zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2018-11-17 03:20:58,744] INFO Starting server (org.apache.zookeeper.server.ZooKeeperServerMain)
[2018-11-17 03:21:03,261] INFO Server environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT (org.apache.zookeeper.server.ZooKeeperServer)
.............
.............
[2018-11-17 03:21:03,345] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory)
[2018-11-17 03:21:03,345] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2018-11-17 03:21:35,434] INFO Accepted socket connection from /127.0.0.1:63924 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2018-11-17 03:21:35,450] INFO Client attempting to establish new session at /127.0.0.1:63924 (org.apache.zookeeper.server.ZooKeeperServer)
[2018-11-17 03:21:35,450] INFO Creating new log file: log.5 (org.apache.zookeeper.server.persistence.FileTxnLog)
[2018-11-17 03:21:35,535] INFO Established session 0x1000238240f0000 with negotiated timeout 6000 for client /127.0.0.1:63924 (org.apache.zookeeper.server.ZooKeeperServer)
D:\Softwares\kafka_2.11-2.0.0\bin\windows>kafka-server-start.bat ../../config/server.properties
[2018-11-17 03:21:30,291] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2018-11-17 03:21:30,777] INFO starting (kafka.server.KafkaServer)
[2018-11-17 03:21:30,777] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2018-11-17 03:21:30,831] INFO [ZooKeeperClient] Initializing a new session to localhost:2181. (kafka.zookeeper.ZooKeeperClient)
[2018-11-17 03:21:35,350] INFO Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT (org.apache.zookeeper.ZooKeeper)
[2018-11-17 03:21:35,350] INFO Client environment:host.name=DESKTOP-RN4SMHT (org.apache.zookeeper.ZooKeeper)
[2018-11-17 03:21:35,350] INFO Client environment:java.version=9.0.4 (org.apache.zookeeper.ZooKeeper)
[2018-11-17 03:21:35,350] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2018-11-17 03:21:35,350] INFO Client environment:java.home=C:\Program Files\Java\jdk-9.0.4 (org.apache.zookeeper.ZooKeeper)
................
................
[2018-11-17 03:21:43,100] INFO [ProducerId Manager 0]: Acquired new producerId block (brokerId:0,blockStartProducerId:0,blockEndProducerId:999) by writing to Zk with path version 1 (kafka.coordinator.transaction.ProducerIdManager)
[2018-11-17 03:21:43,138] INFO [TransactionCoordinator id=0] Starting up. (kafka.coordinator.transaction.TransactionCoordinator)
[2018-11-17 03:21:43,154] INFO [TransactionCoordinator id=0] Startup complete. (kafka.coordinator.transaction.TransactionCoordinator)
[2018-11-17 03:21:43,154] INFO [Transaction Marker Channel Manager 0]: Starting (kafka.coordinator.transaction.TransactionMarkerChannelManager)
[2018-11-17 03:21:43,200] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
[2018-11-17 03:21:43,235] INFO [SocketServer brokerId=0] Started processors for 1 acceptors (kafka.network.SocketServer)
[2018-11-17 03:21:43,238] INFO Kafka version : 2.0.0 (org.apache.kafka.common.utils.AppInfoParser)
[2018-11-17 03:21:43,238] INFO Kafka commitId : 3402a8361b734732 (org.apache.kafka.common.utils.AppInfoParser)
[2018-11-17 03:21:43,238] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)
Note: Should start the zookeeper prior to Kafka server.
You can see the logs, we have successfully started the Zookeeper and Kafka servers.
Then why can’t we create a simple topic and try to exchange some messages between producer and consumer let’s start to do that
D:\Softwares\kafka_2.11-2.0.0\bin\windows>kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic items-topic
Created topic "items-topic".
D:\Softwares\kafka_2.11-2.0.0\bin\windows>kafka-console-producer.bat --broker-list localhost:9092 --topic items-topic
>hai chandra
>hello
D:\Softwares\kafka_2.11-2.0.0\bin\windows>kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic items-topic --from-beginning
hai chandra
hello
If you see these messages on consumer console, you all done. Then you can play with producer and consumer terminal bypassing some Kafka messages.
Apache Kafka quick start
Happy Learning 🙂

Spring Boot Kafka Producer Example
Spring Boot Kafka Consume JSON Messages Example
Setup/Install Redis Server on Windows 10
How to install Apache Kafka on Ubuntu 18.04
Install Mysql on Windows 10 Step by Step
Install Apache Solr on Windows 10
Sending Spring Boot Kafka JSON Message to Kafka Topic
How to Install Ant on Windows 10
How install Python on Windows 10
How to install Elasticsearch on Windows 10
How to install PuTTY on windows 10
How to install RabbitMQ on Windows 10
How to install Gradle on Windows 10
How to install SOAPUI on Windows 10
How to Install Git windows 10 Operating System

Spring Boot Kafka Producer Example
Spring Boot Kafka Consume JSON Messages Example
Setup/Install Redis Server on Windows 10
How to install Apache Kafka on Ubuntu 18.04
Install Mysql on Windows 10 Step by Step
Install Apache Solr on Windows 10
Sending Spring Boot Kafka JSON Message to Kafka Topic
How to Install Ant on Windows 10
How install Python on Windows 10
How to install Elasticsearch on Windows 10
How to install PuTTY on windows 10
How to install RabbitMQ on Windows 10
How to install Gradle on Windows 10
How to install SOAPUI on Windows 10
How to Install Git windows 10 Operating System






charan
March 25, 2019 at 10:29 am - Reply 

Thank you very much chandrashekar, Followed your post as it is and it works like a magic . 🙂 saved lot of time.









NRaj
June 11, 2019 at 4:16 pm - Reply 

Thanks for your post , it was very helpful for me as a beginner my gray area is for topic creation zookeeper is used and producer brokerlist and consumer bootstrap is used if these difference /important of using this is explained it would be  helpful even more









Roshan
June 28, 2019 at 8:29 pm - Reply 

Thanks Chandrashekhar for this detailed post on installing Kafka components on windows 10 machine. If Message Producer (code) and Consumer (code) are running from localhost, then the messages are circulating correctly. But when I run Producer sample code from another machine (other than kafka server hosted machine)  then you need add below line in the server.properties file and restart the kafka server, otherwise message doesn’t reach to kafka instance. 
advertised.listeners=PLAINTEXT://<>:9092
I used .net code with Confluent.Kafka [Confluent’s .NET Client for Apache Kafka] for creating sample.










charan
March 25, 2019 at 10:29 am - Reply 

Thank you very much chandrashekar, Followed your post as it is and it works like a magic . 🙂 saved lot of time.




Thank you very much chandrashekar, Followed your post as it is and it works like a magic . 🙂 saved lot of time.





NRaj
June 11, 2019 at 4:16 pm - Reply 

Thanks for your post , it was very helpful for me as a beginner my gray area is for topic creation zookeeper is used and producer brokerlist and consumer bootstrap is used if these difference /important of using this is explained it would be  helpful even more




Thanks for your post , it was very helpful for me as a beginner my gray area is for topic creation zookeeper is used and producer brokerlist and consumer bootstrap is used if these difference /important of using this is explained it would be  helpful even more





Roshan
June 28, 2019 at 8:29 pm - Reply 

Thanks Chandrashekhar for this detailed post on installing Kafka components on windows 10 machine. If Message Producer (code) and Consumer (code) are running from localhost, then the messages are circulating correctly. But when I run Producer sample code from another machine (other than kafka server hosted machine)  then you need add below line in the server.properties file and restart the kafka server, otherwise message doesn’t reach to kafka instance. 
advertised.listeners=PLAINTEXT://<>:9092
I used .net code with Confluent.Kafka [Confluent’s .NET Client for Apache Kafka] for creating sample.




Thanks Chandrashekhar for this detailed post on installing Kafka components on windows 10 machine. If Message Producer (code) and Consumer (code) are running from localhost, then the messages are circulating correctly. But when I run Producer sample code from another machine (other than kafka server hosted machine)  then you need add below line in the server.properties file and restart the kafka server, otherwise message doesn’t reach to kafka instance. 
advertised.listeners=PLAINTEXT://<>:9092
I used .net code with Confluent.Kafka [Confluent’s .NET Client for Apache Kafka] for creating sample.