The identity crisis: “Testing costs me too much time for nothing?”
By: Faruk Mustafic
Tags: Software Testing, Startup Lessons, Software Engineering, Simulation, Test Driven Development

It can be a controversial topic, but what I know for sure is that every day at least I have a method for approaching the problems I face. This is a sufficient benefit for me on its own. Though, is it worth the price? Let me guide us through the process of trying to find out.

The story begins with the my first green passing test suite.

At first, it all seemed simple: It is true that with tests, some things take longer to implement initially. On the other hand, by first writing tests, I wrap my head around a problem. Besides, I am counting on the long-term savings I am making by introducing less technical debt and preventing regressions from occurring when refactoring or otherwise changing code. And dem gainzzz are real long-term, right? … Right? Anybody?

The other day, I asked myself if this was really true. There are those moments when you start questioning even the most elementary premise you go by. I mean, I do believe it is, but can I somehow quantify it? Or at least show to myself that it makes some sense? Once I started thinking about it, I felt more and more uncomfortable not having the answer and blindly following the method without evert trying to analyze it a bit deeper. At least in this way.

So finally one afternoon, I came up with an idea:

Can I make a simple mathematical model that simulates what would happen with and without using some method that initially takes longer, but reduces technical debt in the long run.

This is not TDD specific, and is applicable to any method with such characteristics. I personally would use this simulation to justify my use of TDD of course.

Please take a note that this model is really crude, the numbers and assumptions are best guesstimates off top of my head. So consider this post as a conversation starter of sort.

If it really does get the conversation started, you can contact me directly via my Twitter handle or in the comments down below.

Sorry to say this, but I have a disclaimer to make: don’t use the data shown here to go to your naggy friend to notify him that you told him so. Do your own research, play with the model and make your own conclusions.

Or even make a new shinny fancy statistical model that will take a lot more factors into consideration. I would be glad in that case, because inspiring somebody to do something has a merit of its own.

Our goal

It is always good to clearly state what you want, not just when running simulations. Repeat after me… We… Want…

We want to measure if it is profitable to invest additional effort in TDD or similar approaches given the potential reductions in cost and time.

Model assumptions

First let’s agree to this: we will examine the assumptions and results and only then go into the mathy meat of the model. Also spaces > tabs if we are on the streak 🤭.

So let’s open up the question that each modeler meditates on:

What does our model assume?

Before dwelling on that, we should briefly introduce some naming.

Let’s call coding methods:

TESTED: i.e. TDD and other methods that involve testing or other practices that improve reliability

i.e. TDD and other methods that involve testing or other practices that improve reliability QUICK: i.e. non-tested OOP code with bad patters sprinkled on top

Below, in Table 1, is a list of our estimates for method attributes. Of course you still don’t know what these attributes means, but you if you kept track so far, you know that the next step is explain each one of them.

Table 1: Comparison of TESTED and QUICK attribute values.

The meanings of the factors are as follows:

factor = Measures how many times longer the method takes to implement the same code compared to the QUICK method. Or in a nutshell, this factor measures how much additional work is added while coding . We assume that each feature takes the same amount of time to build (unrealistic I know, but let’s go one step at a time 👶).

= Measures how many times longer the method takes to implement the same code compared to the QUICK method. Or in a nutshell, this factor measures how much additional work is added . We assume that each feature takes the same amount of time to build (unrealistic I know, but let’s go one step at a time 👶). debt = This one might seem complex, but it is entirely natural and logical. Just follow along. It measures how much more time each new feature or a change takes just because of the effect of technical debt . So for example, if debt=0.01 this means even if we add a new feature of the sample complexity as the previous one, it will still take 1% more time to implement. This happens for various reasons, and one of the best known ones is the so called spaghetti code. The more you add to it, the more time you need to spend locating the place where the new feature or change should reside.

= This one might seem complex, but it is entirely natural and logical. Just follow along. It measures how much more time each new feature or a change takes just because of the effect of . So for example, if this means even if we add a new feature of the sample complexity as the previous one, it will still take more time to implement. This happens for various reasons, and one of the best known ones is the so called spaghetti code. The more you add to it, the more time you need to spend locating the place where the new feature or change should reside. old_bugs = The percentage of time spent on fixing the old bugs. These are bugs that were introduced into the system with some of the previous features and not the ones being added at the moment. Or even the bugs re-introduced again because we didn’t have tests covering those cases.

= The percentage of time spent on fixing the old bugs. These are bugs that were introduced into the system with some of the previous features and not the ones being added at the moment. Or even the bugs re-introduced again because we didn’t have tests covering those cases. new_bugs = We all know the feeling, we spend some of our time chasing bugs caused by edge cases we forgot to cover when hastily trying to bring a new feature closer to production. There is a saying Festina lente in Latin meaning Make haste slowly. Sometimes, by trying to be faster, you end up being slower. With this attribute, we measure the percentage of time spent fixing this type of bugs and glitches.