Mentorship via Debugging: How to Turn a Code Bug Into A Learning Experience
By: Tyler Long
Tags: Engineering, Guild Education, Mentorship, Debugging, Extra Credit

⛰ 🐛 🛫 🛬 🐛 🏖

Here at Guild, we like to foster a welcoming and inclusive engineering environment. In this series we will explore some ways to be an effective mentor as well as an effective mentee. For this segment, let’s use bug hunting as a tool for mentorship! Bug hunting is an invaluable skill, and it becomes an even greater challenge when you are dealing with distributed systems.

Bugs are a part of all systems, and tracking them down is a part of our jobs as developers. But when a bug is no longer a single fly buzzing around a program and becomes a fly stuck in a web of systems, debuggability is more cumbersome and tedious. In order to successfully hunt it down, we need to enhance our understanding of what it means to trace bugs through systems.

Photo by Guillaume de Germain on Unsplash

A systems model is an essential tool for tracing complex problems. Being able to visualize, either mentally or with a diagram, provides a “big picture” context that allows you to identify spaces where bugs can hide. Teaching newer engineers how to trace bugs is a great way to build out mental models for your systems and to level up people on your team. Let’s think of an example.

Imagine that a user reports that they are having trouble logging in. A likely instinct is to check the authentication code. This a logical first assumption, as that code is directly responsible for the actions taken by the user in order to access your system. In a monolithic architecture, we could trace the call stack through one system; but when services are abstracted and microservices adopted, we need a mental model in order to properly map out what may be going wrong. We need to know how the code we think is responsible for the bug is interacting and affecting other code and systems around it.

Let’s use a hypothetical authentication flow in which we have many applications, as well as many services driving the data and logic behind these applications.

At the highest level, this happens:

User visits a login page, enters credentials, and authenticates into Application A.

If we dig slightly deeper, we understand that the process looks more like this:

Code is run to validate that the user exists in our database, passwords match, and the user has all the data we need to establish a connection into Application A. The validation returns true, so grant the user a JSON Web Token (JWT, pronounced “JOT”), establish a session, and carry on with your day.

But thanks to our mental model, we know that reality looks much closer to this.

User visits a login page, enters credentials, and authenticates into Application A (So we have a Single Page App that renders the login form, and handles all client-side validations. We can call this Service One) Service One posts data to a backend service to look up the user and validate their credentials (So, we now have another server-side application handling database lookups and validations as well. We call this Service Two) Service Two does some querying, finds the user, formats the User object into JSON, and passes it back to Service One. Service One receives this data, formats it into a JWT, and hands it over to Application A. But wait! Application A needs some additional data in order to fully render an experience for the user! Application A then queries Service Three, which finds the missing data and passes it back to Service Two, which then re-formats the JWT, and then the JWT is passed back to Application A. Finally we are good to go!

Phew. That’s a lot of services doing a lot of work for an authentication flow! Without an understanding of how all services interact and work together, it’s much harder to determine the root cause. Sure, we can keep making educated guesses about where to target our search, but with a clear mental model of the system, the task becomes less daunting. If you were part of the initial buildout of this system, or helped in architecting it, finding a bug may be much easier; however, imagine what this feels like as a brand new person on the team, or a junior engineer just starting out… it’s confusing!

In a way, I think of debugging as a git bisect of our system — that is, let’s start with the outlying layers and drill inward from there. With a git-bisect we can use a binary search in order to find the commit that introduced the bug. In a system-bisect we can utilize a binary-like search to find what system contains the bug. The mental concepts are the same; only the mechanisms are slightly different. And having a model of our systems allows us to bisect more effectively.

Effective bisecting can help get around an age-old truth about engineering: what you may think is obvious is not, in fact, obvious. Yes, sadly, your assumptions after months or years of working in a system can sometimes leave out important and relevant details for unearthing your bug. So, walk through the basics as part of your process, and train teammates to do the same.

Now, when debugging, context is king. If the engineer debugging this issue has been around for a while, or helped architect some of the system, it is much easier for them to hold on to a mental model while going through the debugging process. Maybe they know that certain users are always missing data Y when they come from a certain population — in fact they have that ticket lingering in the icebox ready to fix this exact issue! So I suggest two things here: