Collect Application Event with Kafka
By: Gökhan Yılmaz Gökün
Tags: Events, Spring Boot, Kafka, Spring, Analysis

The subject I will talk about in this article is application event analysis, which will be a data collection technique that you can use in all your applications in general. With this dataset, you can perform detailed analysis, live dashboards, or live data analysis.

Creation of thread-local event object

All requests coming to our application are captured at the application layer with the interceptor pattern and event analysis begins at this stage. All the distinguishing data on the HttpServletRequest for the requesting client is taken at this stage and added to the event data.

As you will notice from the code sample above, we keep this event data as thread-based variables. This variable is created at the point where the request comes, and it is sent as a Kafka message after the end of the request. Below you can see the structure of the ThreadLocalEventContext class defined as thread-local.

As you can see in the above code snippet, the event object created as thread-local will create a memory leak if it is not deleted after use. Therefore, it should definitely be deleted.

Usage of a thread-local event object

Now it’s time to use the thread-local ApplicationEvent object we created. The structure of the ApplicationEvent object can be created entirely according to your application. You can put all the data you want to analyze on this object. In the use phase, you can access the ApplicationEvent object from any part of your application as follows. After accessing, you can set any data you want on this object and use it in your analysis phase.

After doing this, the data you set on the ApplicationEvent object will be stored for use on this object until the thread ends.

Now we created the ApplicationEvent object we imagined and put all the data we want to analyze on it. Now it’s time to send this data to the queue structure as a message with Kafka.

Finalize thread-local event object and fire to Kafka

We added all the data we wanted to the ApplicationEvent object on the application and came to RequestInterceptor, where we started again. At this stage, we will add our response data to the ApplicationEvent object and make this object can be sent to Kafka.

If you wish, let’s talk about how we can set up the Kafka producer structure in our application.

Kafka Producer Integration to Spring Application

First of all, we define an interface called KafkaProducer and define the functions that this producer can perform. We will only add the fire function, you can add other functions according to your needs. Our KafkaProducer interface is as follows.

Our KafkaProducerService structure, which will implement this interface, is as follows. We used FinalizableDelegatedExecutorService, which is self-terminating in order not to be affected by the current thread during sending messages to Kafka. Thanks to this structure, we sent the Kafka message we fired and forget. This structure handles and ends the rest.

Configuration of KafkaProducer

As I mentioned before, we are doing Kafka producer integration via SpringBoot. While doing this integration, we do the configuration definition together with the application.yml files and spring boot configurations. We use the configuration structure and the usage of configs as follows. By testing the values in these examples, you can find the optimum values suitable for your own structure.

Configurations:

KafkaConfiguration class,

After all implementations and configurations, our Kafka producer structure is now ready for use.

Usage of KafkaProducer

In our RequestInterceptor class, I mentioned earlier, there are two methods, preHandle, and afterCompletion. These methods come from the AsyncHandlerInterceptor interface, which belongs to the spring. In the preHandle function, where the request comes first, we created and added the information of the request to ApplicationEvent, and we started the structure by adding it to the thread-local context.

We can enrich the ApplicationEvent object with HttpServletResponse in the afterCompletion method, which is the last place the application comes before giving a response to the client. For example, the response’s StatusCode information can be given. Let’s add this data and then send this completed ApplicationEvent object to Kafka as a message with KafkaProducer.

As you can see, we can use KafkaProducer by injection into the RequestInterceptor class. In the afterCompletion method, we called the fire method belonging to KafkaProducer and forwarded our ApplicationEvent object to the relevant Kafka topic via KafkaProducer with the help of the configs we previously determined.

As a footnote, I would like to remind you again that after sending the ApplicationEvent object as a message using KafkaProducer, you need to delete the structure we defined as thread-local by calling the ThreadLocalEventContext.clear () method. If not deleted, it will create a memory leak.

Thanks to this structure, you can create all the data you want to gather and analyze in your application in an analytical structure and send them over the topic you have determined with KafkaProducer.

After this stage, it is necessary to develop a structure that can read the messages on the topic to which we send the message and can send a data source by processing it in the desired structure. At this stage, you can transfer to various data sources by using the Kafka connector. Or you can write your own Kafka consumer.

In my next article, I will talk about how we did it, how we analyzed these messages with which structure, and which data source we sent.

When the next article is ready, I will add the relevant link here. You can continue the implementation with that article.