Logging as Your Painkiller. The Ultimate Guide to Great Logs
By: Anton Kazakov
Tags: Software Development, Logging, Software Engineering, Software Architecture

How to make your logs a painkiller

Pain is a fundamental mechanism nature uses to teach all living things what to do and what not to do in this world full of unexpected dangers. It tells an animal when to halt and back off, or, at times, when to run for its life. It triggers an immediate hormonal response, preparing the animal’s body to the drastic change of circumstances. Being a clear, fast and efficient communication channel, pain has done lots of good throughout millions and millions of years of its existence.

However, for us, humans of the modern society, pain is no more a necessary tool for learning things. I think many of us, software engineers, tend to forget this more often than we should :)

We still bear certain amounts of app design-driven pain daily and consider it normal. Because we have no time to fix the situation, of course. And so often this is especially true for logging.

We all learn at some point (through pain and suffering, obviously :) how important the logs are for our apps. Yet, again and again, we actually remember about it only once an outage hits us and the logs… let’s just say, they somehow aren’t helping much.

Let me give you an example.

Unsuccess story

A few years ago, my team and I got to investigate an error that was happening on one of the promotional pages of ours. The page was crashing with a server error from time to time. To make things more stressful, the promotional activity for this page could be stopped only partially, and that meant we were losing money every minute of the outage. So my team started investigating that very second we got the report. We tried reproducing the error ourselves, failed, then checked error rates, only to be able to state: the issue was happening for some users in some cases with no apparent pattern. And that’s it. A bad set of symptoms to start with.

We of course went straight into our log collection system to look at the first application layer logs. They said clearly:

Call to the Service X failed.

Well, cool. Let’s go to the logs of the Service X then, we thought. But wait… Where are they? They weren’t in any of the places we could reach over a link, neither were they, obviously, in our log management software. Well, well.

Service X was one of those components of our microservices-based setup that no one remembers about. They just sit in some unknown server and silently work… until they don’t. This seemed to be exactly that kind of case.

In the next fifteen minutes we were able to find where Service X was deployed, how to get access there, and eventually landed in the folder where we expected to find its logs. But dang! They weren’t there either!

We needed fifteen more minutes to scratch our heads, throw some ideas in, go through a few try-and-fail iterations till someone found the repository of that service and figured out what framework it was built with. The new idea struck us all the same instant: the logs must be in this framework’s default logging folder! Indeed, they were. But they were poorly formatted and thus, incredibly hard to browse through.

After a few more minutes of some power-grepping, we understood that this thing absolutely needs to be reproduced to make it easier to spot in this stream of unknown log lines. We called the marketing folks and asked to refresh their broken page. The error landed in the logs. We found it almost immediately. It went:

An error has occurred.

My colleagues and I got furious :) After all the time we spent looking for this clue, it only made things more complicated. And we were still burning some marketing money every minute of this outage…

Pains caused

Just to wrap up the story above: we figured it out, of course. It took much more effort than it should have, but we fixed the issue and restarted the promotional activities in some two hours from the time we got the first report. However, it didn’t go unnoticed and free of charge. We lost a few things, to say the least.

Our own energy losses were the most apparent. An unexpectedly high issue complexity paired with higher stakes (paid promotion running for nothing, direct business impact, anxious people from marketing standing behind our backs…) drained our energy reserves, and we ended up reducing our productivity for a few days after this rush under pressure.

Money loss was also quite clear. Not only the promotion itself got impacted, but the revenue from it, too. Not good.

Confidence and trust loss was much less obvious, yet much more long-lasting. Not for one month after this incident did our marketing team ask us to spend some extra time on testing new releases, with them included in the process. We, ourselves, switched to a more hesitant deployment approach for a few months, too. The tasks started piling up in the pre-release stage for a little longer, we started overspending time on the regression tests, ‘just to be sure’. This effect likely cost us noticeably more than the direct loss of money during the incident.