A DigIO tech assessment of Cloud Monitoring platforms
By: Michiel Kalkman
Tags: Cloud, Monitoring And Evaluation, Monitoring, Tech, DevOps

Our reference application contains a wide variety of technologies. This will get messy.

Out-of-the box this application is configured to emit visibility data (events, metrics and tracing) and systems for capturing it. This allows us to establish an unbiased baseline.

Out of the box visibility

Pre-flight planning

Our target environment for this assessment was Google’s Cloud platform, GCP, and its managed Kubernetes service GKE. We provisioned a dedicated cluster for each vendor and tracked all modifications made for each in corresponding local version control repositories with git.

For load generation we chose k6. Evaluating load generation tools was not part of the project, but we did compare options before selecting k6. It meets all of our requirements (headless operation, fully script-able, performant) and is used by several teams across the company.

Now we have a platform, an application, the ability to generate load and baseline visibility.

Data inventory and mapping

Preferred division of responsibility for visibility data

We draw a clear boundary between Provide and Collect.

In an ideal scenario our application is responsible for generating all visibility data and making it available (Create and Provide).

There are two main parts to this.

Firstly, this data is either the direct or indirect result of business requirements, making it part of the development effort. Using third-party tools to extract this data after development is time-consuming, error-prone and subject to drift from the requirements as the application changes.

Secondly, generating the data and making it available requires execution of code that operates in the security domain of the application. Introducing third-party code for this complicates operations and makes it much harder to get the application across a security review.

From the monitoring side, the functions of collecting the visibility data, moving it somewhere else and storing it, have nothing to do with the required functions of the application, which are all tied directly to business value. Expending limited engineering resources on custom monitoring solutions is a delivery risk most projects can do without.

A benefit of having this separation is that the further development of the application does not have to take into account the solution being used to monitor it.

Another benefit is that is becomes possible to switch monitoring solutions fairly easily because we only have to reason about the Provide/Collect interface.

In reality, this ideal is never quite realised, but the closer we stay to it, the better our position is in relationship to unknown future changes.

Visibility data inventory — Where we start

This is our starting position. The blue area is our responsibility, the red area is the responsibility of the monitoring platform. Here our blue area crosses the Provide/Collect boundary. This is caused by us providing both the Kubernetes infrastructure as part of the solution and pushing the tracing spans into Jaeger from the application tier. Ideally we’d see everything under Create and Provide being blue and the Collect, Transport and Store space being red.

Where we want to be

Instrumentation

Remember earlier on when we talked about things getting messy? This is where that starts to happen. We want to divide the responsibilities cleanly between our solution and the monitoring stack, but this rarely goes entirely according to plan.

One provider we evaluated did not have the ability to receive or collect OpenTracing spans, so in order to have a level of tracing visibility we instrumented an APM solution, which can mean runtime modification (adding an agent to the Java classpath) or importing third party libraries, changing code and rebuilding the applications (NodeJS, Go).

Another provider would accept OpenTracing spans (HTTP 200/OK), but fail to register them upstream. We had to go down the APM route to get tracing coverage there too.

Where we often end up

Infrastructure tends to be easier to integrate with default tools and settings but Application code is where the business logic resides. This results in a much greater variety and complexity.

The red lines moving into the blue space on the left reflect this and indicate a tight coupling of the vendor tooling with the application code. This presents clear risks for both present and future changes.

For example, one of the Java APM agents provided has support for Java 8, but no higher version. If we want to move to, say, Java 11 we have to either drop the APM capability or keep the application on an outdated JVM.

Another example, the vendor libraries we’re adding become part of our software supply chain, have to be moved through our CI/CD pipeline and pass the same security scans as all our other code. Several monitoring components used in this assessment were not available via public channels and required separate downloads. Automating downloads in a CI/CD pipeline requires bespoke code.

Usually all the code that goes into our applications is completely accessible at the source level. This happens via external libraries being open source on the one hand and full access to the internal application code on the other.

Introducing externally generated executable code into this mix not only makes the CI process more complex and fragile, it also makes it harder to perform useful automated security analysis.

Lesson learned — No such thing as a trivial third-party addition.

Load testing

We ran into upstream rate-limiting issues, monitoring pods exhausting resources and a variety of other failure scenarios.

Our forwarder being rate-limited

All of these issues were ultimately resolvable and most were related to the nature of cloud and SaaS based solutions.

Upstream rate limiting turned out to be a result of PoC (Proof of Concept) sizing on the monitoring platform, it was not set up for the volume of data we were sending. Crashing monitoring pods were likewise related to resource allocation.

Lesson learned — Assumptions about traffic bursts will be false.

Today we learned that service maps are very pleasant to look at

Another recurring theme was that monitoring tools need monitoring too. Almost all are self-monitoring and that aspect is highlighted as a feature, which is indeed very convenient.

However, when the upstream connection disappears or your license expires, you’re left flying blind. A fallback option for collecting the visibility data in the Provide stage locally is always a good idea.

Lesson learned — Diagnostics through self-monitoring will not be available when you need it

This just in: Cloud resources cost money

We significantly underestimated the amount of the resources required to run monitoring tools in the cloud. Most unexpected was the amount of compute used by traffic optimisation to lower network utilisation. It seems that costs don’t disappear and all ultimately end up somewhere.

Lesson learned — Resource consumption is hard to control

Requirements and value

So far we’ve focused on obstacles and lessons. It’s probably a good time to point out that monitoring platforms add a great deal of value. Building bespoke monitoring solutions from open source components is time-consuming and might not be the best way to burn your engineering budget.

Every integration has some awkward aspects, but those can be managed. Before you select a solution, you have to figure out what you and your stakeholders want from it.

This gives you a list of requirements that can be categorised according to their respective visibility products. Those products are constructed from metrics, events and tracing data.

Example uses for Tracing data

Once you know what visibility data you have you can connect estimate what the cost is of transporting that data to a remote platform. Knowing what your requirements are and what data you have, you can determine whether you have enough visibility data to meet those requirements.

Putting all of those together we can now work on evaluating whether a platform is capable of giving us access to our data in the form of the various visibility products we require.

Partial requirements breakdown

Wrap up

We didn’t get to spend much time building reports, dashboards and alerting in this assessment. Most of our time was spent connecting to the different monitoring platforms and running our load scenarios. The platform defaults with minor tweaks gave us most of what we needed.

We learned a lot about the messy details and risks of integrating with monitoring platforms, assembled a repeatable process for evaluations, presented our findings back to our colleagues in DigIO and generally had a good time.

Looking forward to the next assessment!

For more information, read up on how tech assessments fit into our strategy for staying current at A DigIO approach to tech vitality.