Reporting and data visualization in Kibana
By: Veronika Rovnik
Tags: Elasticsearch, Programming, Data Analysis, Big Data, Data Visualization

Introduction

In this article, I’d like to show the functionality of Kibana based on my personal experience. My purpose is to give a high-level overview of Kibana’s capabilities and analyze the need in the custom plugin for pivoting and reporting. In the following articles, I hope to have an in-depth exploration of Kibana and cover the data visualization process in more details.

What is Kibana

With over 11k stars on GitHub, Kibana steals the hearts of developers all around the world and holds a solid place of the best platforms for visualization of Elasticsearch data for many years. And not without a reason.

So what is it about Kibana that makes it a must-have tool for Elasticsearch?

Its main purpose sounds simple yet it’s indeed mighty:

Kibana is designed to help you understand your data better by providing a single interface that makes interaction with the Elastic Stack easy and time-saving. Kibana does a lot of heavy lifting for you, including querying Elasticsearch for the data through the REST API. In this way, it eliminates the need for manual writing queries.

better by providing a single interface that makes interaction with the Elastic Stack easy and time-saving. Kibana does a lot of heavy lifting for you, including querying Elasticsearch for the data through the REST API. In this way, it eliminates the need for manual writing queries. Another feature you’ll love is the fact that Kibana is 100% open-source — its source code is available on GitHub repository and everyone can contribute to its development. Due to this, the product is actively updated, maintained and enhanced.

— its source code is available on GitHub repository and everyone can contribute to its development. Due to this, the product is actively updated, maintained and enhanced. Moreover, the community is incredibly strong and supportive — you can sense it on the Kibana Discuss Forum.

Visualization capabilities

Let me give you a short overview of the visualization functionality.

The magic starts as soon as you import your data into Kibana.

At your disposal, there are multiple core visualizations such as area, pie, line and bar charts, histograms, sunbursts, heat, region, and coordinate maps, gauges, data tables, tag clouds and many other.

Controls are worth special attention — with them, you can add interactive inputs (dropdown menus and radio sliders) and filter the content of a dashboard in real time.

In addition to the basic visualizations, Kibana also supports Vega — a high-level grammar that allows the rapid building of custom visualizations.

If you are interested in analyzing the time series data, you can take advantage of the functionality that Timelion provides. Timelion is a special type of visualizer that takes your raw time series data and presents it in a way so as to help you get actionable insights from your data.

As soon as you’ve defined an index pattern and chosen a visualization, you can apply metrics aggregation and bucket aggregation to categorize the documents that match certain search criteria.

One may be impressed by the multitude of the Y-axis aggregations for all tastes and purposes, especially statistic ones: sum, average, min, max, count (unique count), standard deviation, median, percentiles, percentile ranks, top hit, and geo centroid.

For the X-axis you can benefit from such bucket aggregations as date histograms, ranges, terms, filters, and significant terms. Along with aggregations, you can divide the data further by applying subsequent sub aggregations.

In addition to the mentioned aggregations, you can define parent pipeline and sibling pipeline aggregations. To learn about all the possible aggregations in details, I recommend reading this comprehensive article.

To visualize the data only from those documents that meet certain criteria, you can add field filters.

There are two options for filtering:

By writing a search query using either the Kibana Query Language (KBL) or Lucene as a syntax.

(KBL) or as a syntax. By clicking elements of visualizations.

The interface for adding filters is really comfy — check it out:

Nothing limits you to write complex filters that are based on multiple fields and conditions.

Sample of visualization

Here is an example of a stacked bar chart which you can create to know the average delays of carriers’ flights to certain destinations:

This visualization is built on the basis of the sample flight data.

Saving results

As soon as you feel satisfied with your results, you can save the created visualization. It’s ready to be added to the dashboard as well as shared with your teammates. Visualizations are completely reusable.

Adding components to a dashboard

Here is an example of a dashboard you can build with Kibana:

You can inspect every visualization by peeking into the underlying raw or aggregated data and even export this data to CSV if necessary.

For completeness of the data analysis process, the resulting dashboards can be converted to tailored reports in PDF and PNG formats.

Personal impression

One of the coolest advantages is that Kibana gives you the freedom to control pretty much every aspect of your dashboard. From the data source that you feed into the visualization to the tuning individual colors of a particular element. Other strong features are aggregation and filtering capabilities.

But especially I like that the elements of the dashboards are totally interactive and configurable. Every component can be customized according to your preference — you can change colors, legends, titles, etc.

Enhanced reporting in Kibana

As a tool for business intelligence, Kibana is perfect. Its interface allows creating a dashboard in minutes and analyzing the data with its help.

Despite having tons of visualizations, it would be great if Kibana had a tool designed solely for advanced reporting purposes. It should give an ability to look at the data from different angles by slicing and dicing it. To my mind, a pivot table suits the best for this task. There is a plugin for Kibana that is easy to install and use with the Elasticsearch data. It’s called Flexmonster. Though initially created as a pivot table component that can be incorporated into any app that uses JavaScript, it can serve as a part of Kibana as well. You can connect it to the Elasticsearch index, fetch the documents from it and start exploring the data.

Pivot table’s features

Flexmonster offers multiple features for effective reporting such as aggregation functions, filters, sorting, and built-in exporting of reports to PDF, Excel and other formats.

As for aggregations, there are sum, count, distinct count, average, min and max functions available.

Filtering options

You can filter the data by members’ names or report filters that keep your report well-organized.