What startups should know about observability
By: Merlin Carter
Tags: Startup Lessons, Monitoring, Distributed Systems, Observability, DevOps

It all depends on your application and business

Stephan: Well, the thing about storage is that it’s relatively cheap. So you can store all your event data on an S3 bucket and retain it for a few weeks in its original format.

And after that, it gets compressed. Even if you need to investigate something further back in time, you can rehydrate that compressed data from storage.

The more difficult problem is deciding what data to collect and observe. Of course, you can get more insights if you have more data.

But you should consider this carefully because it can have a huge impact on costs. You could also choose to have different levels of detail for different systems.

On a production system, maybe you just log critical signals like application errors or warnings. And maybe you only enable debug messages for a specific part of your application — a part that you need to understand better in production

Personally, I prefer a more differentiated approach rather than logging a shitload of information. But there are pros and cons on both sides.

What is also interesting for me is tracking product-related metrics.

For example, you can use observability tools to look at the timing of different events and create a metric such as “successful checkouts” or “successful or failed logins”.

The idea behind this is, that you can look at these metrics in a specific time frame and compare them to the same time frame last week.

That way, you can look for specific trends, and see how your application is performing.

Merlin: That’s interesting — what you’re talking about is a business metric. Checkouts have a direct impact on your bottom line. So you pay closer attention to metrics like that, rather than database requests, or I/O operations?

Stephan: Your focus should really depend on what you want to achieve. Certainly, developers should and will still be interested in the more technical metrics.

But these product metrics can also give you an indication that your application has a problem and it’s probably not a technical issue.

For example, let’s say you’re implementing a new feature and want to see any impact that it might have on the user experience.

You could track the number of successful logins versus the number of failed logins.

If you see a spike in the number of failed logins, you might have to roll back the feature because it’s really affecting your users.

This is a reminder that you’re running a business and not just a system.

An e-commerce business doesn’t care about I/O operations — it cares about sales and purchases.

So, I would say that developers need to monitor both the technical data points AND the business KPIs.

Merlin: OK, so there’s no universal list of data points that all developers should monitor. You really need to think about the metrics that are important for your specific application and business.

Stephan: Exactly, it all comes down to the problem that you’d like to solve.

If we look at a concrete example.

We’re supporting an e-commerce company that is migrating their application from bare metal servers to a Kubernetes setup in the cloud.

As this happens with production systems there is a lot of business risk involved. In that case, we decided to track successful checkouts next to a couple of other metrics.

By being able to compare the number of successful checkouts from the current day with the ones from last week or yesterday gave us a pretty good safety net that prevented us from not running into any hidden problems.

And as we had only limited information available, that metric was easy to track because we just looked at the number of users who saw the “thank you” page and the end of the checkout process.

These kinds of tools also help you to filter out any misleading data, such as cases when users fail to check out because of an ad blocker or similar.