Building Spring Boot Microservices , Monitoring with prometheus and grafana and log aggregation using ELK stack: Part III
By: Firas Messaoudi
Tags: Loggin, Kibana, Spring Boot, Logstash, Elasticsearch

Building Spring Boot Microservices , Monitoring with prometheus and grafana and log aggregation using ELK stack: Part III Firas Messaoudi Follow Feb 20 · 6 min read

ELK

In the previous parts we maneged to create 3 springboot microservices and monitor them with prometheus and grafana.

In this part we’re gonna focus on the Log aggregation and analysis.

Why it is so important ?

Logs are essential informations, allowing us to better understand, analyze and diagnose certain malfunctions, as they can help us follow and trace the different actions carried out by the actors of our system. Only, it becomes very complex to exploit this critical data with so many applications, equipment, services, systems and managed elements in an information system.

log analysis

As a solution for this problem, we’re gonna use ELK stack, one of the best and efficient tool for log aggregation and analysis.

So what is Elk stack ?

Composed of Elasticsearch, Logstash and Kibana, ELK is a suite of three complementary open source projects. This trio of tools will make it possible to display an overview to the system administrator while offering him a quick graphical analysis of the elements he is in charge of.

ELK

1-Elasticsearch: Elasticsearch is a real-time distributed and open source full-text search and analytics engine. It is accessible from RESTful web service interface and uses JSON documents to store data. It is built on Java programming language and hence Elasticsearch can run on different platforms. It enables users to explore very large amount of data at very high speed.

2-Logstash: Back then, logstash has been used to process logs from applications and send them to Elasticsearch. But , since then logstash has evolved into more general purpose tool that makes it a data processing pipeline.

The data collected by logstash will be processed and shipped of to one or more destinations (Elasticsearch, kafka, e-mail…).

3-Kibana: Kibana is basically an analytics and visualization platform, which lets you easily visualize data from Elasticsearch and analyze it to make sense of it. You can assume Kibana as an Elasticsearch dashboard where you can create visualizations such as pie charts, line charts, and many others.

Global Architecture

The following is the architecture of ELK Stack which shows the proper order of log flow within ELK. Here, the logs generated from various sources are collected and processed by Logstash, based on the provided filter criteria. Logstash then pipes those logs to Elasticsearch which then analyzes and searches the data. Finally, using Kibana, the logs are visualized and managed as per the requirements.

elk architecture

Elk with microservices

All our microservices will push their respective logs to Logstash, which will use Elasticsearch to index them. The indexed logs can be consumed later by Kibana.

We will use Logback as a logging framework.

Logback offers a faster implementation , provides more options for configuration, and more flexibility in archiving old log files.

Setting up ELK with spring boot microservice

1-Installing Elasticsearch

Download Elasticsearch zip file from https://www.elastic.co/downloads/elasticsearch

Unzip it then run it.

To make sure ecerything works fine open the browser on : http://localhost :9200. You must see something similar to this cluster informations :

2-Installing Kibana

Download Elasticsearch zip file from https://www.elastic.co/downloads/kibana

Unzip it then run it.

Open up Kibana in your browser with: http://localhost:5601. You will be presented with the Kibana home page.

3-Installing & configuring logstash

Download Elasticsearch zip file from https://www.elastic.co/downloads/logstash

Unzip it.

A logstash configuration consists mainly on 3 stages:

1. Input

The input section in the configuration file defines the input plugin to use. Each plugin has its own configuration options, which you should research before using.In our cas it will be a json flow fetched from the logback of our microservices and we will be using the tcp plugin wich will listen to any logs sent.

2. Filtre

The filter section in the configuration file defines what filter plugins we want to use, or in other words, what processing we want to apply to the logs. Each plugin has its own configuration options, which you should research before using.

3. Output

The output section in the configuration file defines the destination to which we want to send the logs to(elasticsearch for example). As before, each plugin has its own configuration options, which you should research before using.

So our logstash file config will be similar to this:

input {

tcp {

port => 7000

codec => "json"

}

} output {

elasticsearch {

hosts => ["localhost:9200"]

index =>"ecommerce"

}

}

Run it

Losgatsh will start and exposes two ports:

9600 logstash api endpoint

7000 tcp input listener defined previously.

4-Configuring logback

We need first to add these maven dependencies to each of our microservices:

<dependency> <groupId>net.logstash.logback</groupId> <artifactId>logstash-logback-encoder</artifactId> <version>5.1</version> </dependency> <dependency> <groupId>ch.qos.logback</groupId> <artifactId>logback-core</artifactId> </dependency> <dependency> <groupId>ch.qos.logback</groupId> <artifactId>logback-classic</artifactId> </dependency>

Then add these config class :

@Configuration public class LogbackConfiguration { private static final String LOGSTASH_APPENDER_NAME = "LOGSTASH"; private static final String ASYNC_LOGSTASH_APPENDER_NAME = "ASYNC_LOGSTASH"; private final Logger LOG = LoggerFactory.getLogger(LoggingConfiguration.class); private final LoggerContext CONTEXT = (LoggerContext) LoggerFactory.getILoggerFactory(); private final String appName; private final String logstashHost; private final Integer logstashPort; private final Integer logstashQueueSize; public LogbackConfiguration( @Value("${spring.application.name}") String appName, @Value("${logstash.host}") String logstashHost, @Value("${logstash.port}") Integer logstashPort, @Value("${logstash.queue-size}") Integer logstashQueueSize) { this.appName = appName; this.logstashHost = logstashHost; this.logstashPort = logstashPort; this.logstashQueueSize = logstashQueueSize; addLogstashAppender(CONTEXT); } private void addLogstashAppender(LoggerContext context) { LOG.info("Initializing Logstash logging"); LogstashTcpSocketAppender logstashAppender = new LogstashTcpSocketAppender(); logstashAppender.setName(LOGSTASH_APPENDER_NAME); logstashAppender.setContext(context); String customFields = "{\"servicename\":\"" + this.appName + "\"}"; <3> // More documentation is available at: https://github.com/logstash/logstash-logback-encoder LogstashEncoder logstashEncoder = new LogstashEncoder(); // Set the Logstash appender config logstashEncoder.setCustomFields(customFields); logstashAppender.addDestinations( new InetSocketAddress(this.logstashHost, this.logstashPort) ); ShortenedThrowableConverter throwableConverter = new ShortenedThrowableConverter(); throwableConverter.setRootCauseFirst(true); logstashEncoder.setThrowableConverter(throwableConverter); logstashEncoder.setCustomFields(customFields); logstashAppender.setEncoder(logstashEncoder); logstashAppender.start(); // Wrap the appender in an Async appender for performance AsyncAppender asyncLogstashAppender = new AsyncAppender(); asyncLogstashAppender.setContext(context); asyncLogstashAppender.setName(ASYNC_LOGSTASH_APPENDER_NAME); asyncLogstashAppender.setQueueSize(this.logstashQueueSize); asyncLogstashAppender.addAppender(logstashAppender); asyncLogstashAppender.start(); context.getLogger("ROOT").addAppender(asyncLogstashAppender); } }

And finally add these properties:

logstash.host= localhost logstash.port= 7000 logstash.queue-size= 512

Now let’s run our microservices and go to kibana and start playing with our index.

In kibana , go to management and click on create index pattern:

We will find the name of the index mentionned in the logstash config file.

Now go to the discover section and you will find the log flow generated by our microservices:

Now you can play with kibana and generate cool dashbords depending on what you need.

The latest version of the project is available in the git repository.

https://github.com/FirasMessaoudi/ecommerce