in this video we will learn how to integrate Kafka into our microservices project and see how to build inventory and microservices using Apache Kafka we'll also integrate uh some best practices like using schema registry and also learn how to write integration test when you are dealing with Apache Kafka and spring boot so without any further delay let's start the video all right so let's start by learning what is Kafka so Kafka is a distributed event streaming platform or a messaging platform uh
00:30
which is used to build even driven uh architecture even driven applications so this is a very widely used um software in the industry right now so whenever you are going to use a messaging platform like uh you will have an option to use apachekafka because it's very reliable it has some feutures like uh High availability so it can have know highly available clusters set up across set up across different availability zones the and also the core future of Kafka is it provides permanent storage of the messages and of course it is very
01:03
scalable and it provides very high throughput of the of messages so so if you're new to if you're new to Kafka I suggest you to go to this cfa.org website and know and explore some explore about Kafka we will mainly Deep dive into how to integrate um Kafka into our project and with spring boot so to integrate uh Kafka with our application so to integrate uh Kafka with our project we we'll be mainly using the spring for Apache Kafka Library so it provides a good abstraction and libraries to integrate with Kafka we
01:39
have some Kafka template which helps us which enables us to easily send messages to the Kafka topic and also Kafka listener annotation which creates a listener which will read the messages from the Kafka topic and helps us to process the messages so we'll also see how to use this Library so if we go back to our architecture diagram you see that our order Service uh will be making the asynchronous communication with the notification service so we'll be integrating so we'll be using Kafka here for the asynchronous communication so
02:11
what I'm going to do now is to enable the Kafka integration in the order service and after that we will create the notification service which is um actually sending out emails to the the customer um by reading the messages from the Kafka topic so now let's open the audit service all right so I'm inside the order service project and the first thing I'm going to do is to install Kafka in our project so for that I'm going to open the doco compost file and I've already added know some uh doco
02:42
compost some services to the do compost uh because it's easy for me to know just explain this um do compost file instead of know you watching me you watching me type all the services one by one into the doco compost file right so what you have to do is uh just go to the blog post which will be linked down in the description section so if you go to the blog post here uh in the first section installing Apache Kafka through Docker you can just copy the whole dock compost file it contains all the necessary know
03:16
uh services to run uh Kafka so if you go back to our intellig you can see that the first service we have is uh zuker something called as zuker so zuker is also an Apache aache project project so this is mainly used to orchestrate uh different clusters of Kafka right so Kafka can uh run in a clustered environment as uh as we saw in the previous um in the pre in the Kafka website it's highly available that means it runs across different clusters so uh zookeeper acts as an intermediary where it manages different clusters right it
03:53
manages the state of different kafa clusters and it has centralizes and centralized storage to manage state of the the Kafka clusters so zookeeper is used mainly as an orchestrator so that's why we need uh zuker so for zookeeper we are using the Image conflent Inc CP zookeeper version 7.5.
04:15
0 and I'm going to call the host name as zuker and the container name as zuker and of course it's I'm also going to expose the port 2181 so this is the port in which uh zuker will be running so I'm going to expose zuker on the same port and finally I have two envirment variables zuker client Port as 2181 and zookeeper tick time as 2,000 milliseconds so this is the configuration for zuker in the latest version of apachekafka there is no need to use uh zuker anymore uh you can use something called as Craft kft um but uh I'm not going to use that right now um
04:54
so just as as an information that you know you can also use um apach Kaa without zookeper so next comes the service uh Kafka itself so I'm going to call this service as broker um instead of Kafka as broker and we are going to mainly use the Image conflent Inc CP Kafka 7.5.0 the version 7.5.
05:19
0 the container name is going to be broker and here I'm going to expose two ports 9092 and 2992 and uh this um Kafka service depends on zuker because um before running uh Kafka it needs uh it needs to it needs to register itself with zuker so that's why we depend on this containers first and we also have some environment variables the Kafka broker ID know it's which is uniquely identifying the the broker and after that we also have to provide the environment variable uh to which service the Kafka should connect for zuker to register itself with zuker so for that
05:58
we have Kafka zuker connect environment variable which is we are providing it as the container name zuker followed by the container Port 2181 finally we have some security configuration like Kafka listener security protocol map so here as we are using just know in the local development I'm just going to provide the security protocol as plain text so or else you can use know SSL and TLS TLS security protocol when you are using it for production and uh we also have Kafka advertised listeners that means these
06:30
are the listeners where Kafka is accepting the connections right so we have two listeners plain text uh on the name broker 2992 and for the plain text host Local Host 9092 so mainly if you are communicating from one container to other so if you want to access Kafka from one container uh from a different container you'll usually use this 2992 port and if you want to use this from know from your Local Host machine you use Local Host 909 2 so that's why we have uh exposed two ports 9092 and 2992 finally we have the Kafka offsets topic
07:07
replication Factor environment variable we are going to set it as one we are going to set it as one and we have another um service here called as schema registry uh I'm going to come to this part uh later so let's um so to to make you understand why we need schema registry first we have to know uh try to develop the logic to to send out messages to Kafka from our order service and then we will come and know uh enhance it with schema registry so lastly we have the service called as Kafka UI which provides a nice UI where
07:42
you can you know view the topics and the messages which are there in the the Kafka topics so for that we are going to use the image Kafka UI from Proctor Labs I'm just going to use the latest image and I'm going to run this um container on the port 8086 internally in the Docker uh in the docker container it's going to run on the port 880 because as we are already using the port 880 I'm just going to map this into a different port 8086 and uh this Kafka UI is going to depend on the broker and the schema
08:14
registry because it needs both of them uh before we run this application so for now we can just remove the dependency to schema registry We'll add this back once we want to know integrate again the schema registry and lastly we have to have the environment variables Kafka cluster's name is local that means we are just going to you know use the local cluster here and the cluster bootstrap servers I'm going to provide the address as broker 29092 because here you can see that the Kafka UI is running in its own
08:46
doer container right and it wants to if it wants to connect to this um Kafka running in the do container so it has to use the name broker followed by the 2992 port right so finally we also have the schema registry um environment variable so I'm going to just comment this out and uh lastly we have this Dynamic config enabled as true this is not really know um necessary for us to go D deep into this configuration this is specific to this Kafka UI project so the we we are just happy by using the defaults here so once this is done uh
09:27
I'm just going to go to the terminal and I'm going to run the docker compose up minus D command again and you can see that the docker containers are all started successfully so if I go to the docker desktop and go to the broker we can see that we don't see any errors uh at least for the Kafka container so let's go back and uh also for zookeeper we don't see any errors looks good so we have installed um in this way we can install Apache Kafka using do compose so again if you need something um from the
10:05
do compose file just make sure to refer to the blog post now let's go ahead and install uh the libraries to enable Kafka integration in our spring wot project so for that I'm going to the pom.xml file and the easiest thing we're going to do is to open the browser and open a browser new tab go to start.
10:27
spring.io and here I'm going to add dependencies make sure to have you select the project Maven and click on ADD dependencies and search for spring for Apache Kafka spring for Apache Kafka and also search for test containers so if you add both of these uh dependencies and click on explore you can see the dependencies for gafka so the first one I'm going to use is this spring Kaa dependency so just copy this configuration here and go back to intellig and paste it somewhere not below the dependency of spring boot
11:06
actuator and the next thing we need is the dependency for spring C cut test I'm going to again paste this below the spring boot star a test dependency and lastly we also need the dependency for CFA test containers so I'm going to just paste this below the test container my SQL dependency right after adding all these dependencies just make sure to click on the maven icon on the top right corner to load the changes so once this is done um our the Kafka spring for Kafka library is downloaded successfully uh on
11:46
your machine so now the next thing we are going to do is to configure Kafka in our spring boot project so for that I'm going to open source main resources folder and inside the application. properties file I'm going to add a section called as Kafka properties so here I'm going to type the property spring.


Q- format this as step by step tutor and finally create 10 question/answser style exercise to explain more technically the detail in the article