Really? I could see all of this ? — The real power of an Observability Platform.
By: Suraj Pandey
Tags: Elasticsearch, Observability, Dynatrace, Elasticsearch Performance, Distributed Tracing

Really? I could see all of this ? — The real power of an Observability Platform. Suraj Pandey Sep 7·6 min read

What you see on the surface is not always the reality. You need the right tool to dig deeper and see for real before you make any conclusions. Now that I have the power of Dynatrace AIOps, I can see things at the code-level and analyze RCAs in minutes, and I no longer need to spend hours searching for logs or curating data for that matter. This biz-tech article is my experience of using an observability platform to uncover any hidden bottlenecks in prod deployments without having to do anything at the data layer, but just focusing on the AIOps tool.

Gain complete insight of your platform quicker than anyone has ever shown you before!

THE WORKLOAD

The Scene

I setup a prod-like logging environment with all the bells and whistles (multi node cluster, replicas, backup, load-balancer, dedicated UI, …). Choosing an open-source Elastic Stack was a no-brainer (you guessed it right! thats what I did in my past life). Load generation via esrally was a piece of cake too.

Btw, this the “workloads” I am trying to monitor could have been any application or platform. I chose elasticsearch just because I see it is a common logging solution for many.

The Drama

It didn’t even take minutes for the load on the logging platform to peak and saturate CPUs. The benchmarking tool (Esrally) was doing its job well.

The Climax

The operational drama was unfolding in my logging platform. The _bulk indexing requests queuing, CPU and IOwait times increasing, the pauses caused by garbage collections, actually everything! Wait… what? GC collections and the pauses? The process restarts? Let’s dig deeper, shall we?

THE OBSERVER

I had to use Dynatrace, not just because I am working for the company, but because of the ease with which I could get complete visibility of the platform with minimal effort around configurations or schemas or anything at the data layer really.

With the OneAgent running in the logging platform (just a single agent to be precise) and collecting everything (full-stack monitoring they say) from host to platform to application to services to code — yes everything, all I needed to do was to go to Dynatrace UI and start navigating.

FOCUS ON INSIGHTS

Dynatrace removed the complexity of dynamic workloads by providing automatic and intelligent observability, without requiring any code changes. This all happens out-of-the-box with the Dynatrace OneAgent, which automatically discovers and maps all applications, services, and infrastructure as well as any dependencies, without configuration or scripting, and without having to know which apps or cloud platforms are running. This provides end-to-end visibility into the running of the application, saving time spent manually identifying any anomalies and allowing teams to get to the root cause of code-level issues quicker. This means full transparency into application data and more time to focus on developing feature-rich applications for your end-users.

Dynatrace presented the top problems in the platform (which happened during the benchmarking/stress test run) so I dug deeper into each of them straight away.

Dynatrace AIOps identified problems for me straight away

Drill Down to do Root Cause Analysis

I wanted to know why the CPU was saturating. To my surprise, it was the pbzip2 that was taking most of the CPU during the deflating of the benchmark files (ref: http_logs, nyc_taxi, etc. from esrally).

ESRally downloads the benchmark and when the files are deflated, the CPU saturates.

I went into digging the RCA for the second CPU saturation. This one was a legit! While Elasticsearch was only taking around 53% of the CPU, there were other processes that were consuming the resources. So in combination, the poor container gave up! The culprit was the docker-desktop (which even had a restart, can you believe that?).

Just looking at the CPU graphs, you’d say its Elasticsearch. But its NOT!

The Dynatrace Platform automatically baselines all performance metrics of the applications its monitoring. When the response times of an application increase beyond the auto-detected baseline, the platform creates an alert with key insights.

AIOps clearly identifies the bottlenecked process that is causing all the issues.

In addition, the Dynatrace platform allows you to drill down into root-cause insights such as the origin of the increase in response time.

All failures are also automatically detected by the Dynatrace platform. In the below example, the Elasticsearch process was in suspension due to a “Long garbage-collection time”, and the Dynatrace platform was able to identify the issue and report the root cause:

Those increase in response times were then poorly reflected in the indexing and searching times, as you can see in the charts below. Elasticsearch is blatantly fast, but due to the suspension (in this particular case), the total search time exceeded 1s.

What did the memory hold that contributed to that long GC cycles? I got into memory profiling and then found out the elements that “survived” vs that were collected. Now you know why Elasticsearch requires higher memory!

I was also interested in knowing which method in the Elasticsearch implementation was doing the _bulk ingestion and how long they took for adding documents into indices.

Code-level hotspot analysis:

The Dynatrace platform automatically identifies methods in your application that are CPU-intensive, so you do not have to dig into the data.

The Dynatrace platform automatically identifies methods in your application that are CPU-intensive, so you do not have to dig into the data. Exception Analysis:

All code level exceptions are logged, along with details into the stacktraces of where the exception occurred.

Besides code-level views, I then got into distributed tracing to see which service calls took the highest response times. Dynatrace PurePaths visualization helped me get into those specific service end-points. I came to know why Elasticsearch monitoring was so busy doing what it was doing.

I also wanted to identify all the top end-points ordered by the number of times they were called and get details into how time-consuming they were. This helps developers immensely in optimizing application usage.

Well, the Dynatrace OneAgent monitors so many technologies (as grouped in the chart below), I could end up writing another PhD thesis : — ) You could certainly go ahead and try Dynatrace for yourself.

Learning experience has never been this much fun before.

Stay tuned for another one next week…

Disclaimer: This article is purely my experience of using Dynatrace platform to learn and share, and should not be considered anything else. Elastic and elasticsearch are products that belong to Elastic.co. I used it for educational purposes only.