3 Lessons Learned On Distributed Systems Design
By: Tuba Kaya
Tags: Distributed Systems, Microservices, Software Engineering, Software Architecture

Photo by Christopher Gower on Unsplash

The first project I designed with microservices architecture style did not go well. I understood the books and articles on the subject. I even had worked as a developer on a large distributed system before. Yet, it was very difficult to apply my understanding to design a project in a different domain.

Looking back at my almost a decade long journey in highly scalable distributed systems design and development, these 3 lessons I learned along the way are the ones that made a difference.

1. Don’t be afraid to try out topology

It takes years of research and experimentation to gain confidence in figuring out the better fitting tactics to achieve desired quality for various systems. You realize the abilities, and more importantly the limitations, of each approach. Not having this confidence can stand in the way of making decisions.

I remember trying to make the perfect choice and without that, not daring to decide. Thinking harder and harder, trying to find what I was missing and why none of the options made all our wishes come true.

Certainly it takes some time to lay down a design idea that seems to satisfy the goals. But if you find yourself not able to find the perfect solution, you should go ahead with the next best thing.

Time and again I experienced that what we came up with, as the components topology and their interactions, didn’t work out as we thought it would. Now I ask myself how we missed such obvious problems in the design. But, we did make the mistake of not realizing even these shortcomings of the design:

forgetting to take into account certain data being meant to be processed together;

calculating the limitations of scalability targets of the storage we chose;

the data we selected would exceed the size limitations of messages by the chosen messaging orchestrator

Even though I learned from these mistakes to make sure to include these in the thought process while designing the components in the hope to get it right the first time, I also learned how possible it is that we fail to foresee the problems that seem obvious after we see them. The actual implementation together with early measurement of system limitations, is the absolute way to realize these mistakes.

Therefore the first lesson for me was to push forward with design choices without being stuck on the desire for perfection. And finding out the flows of the design as early as possible, leads us to the second lesson...

2. Test early to know the limitations

Reasons to choose a microservices architecture style are many fold. But the most emphasized of them all is scalability. Compared to measuring maintainability, measuring scalability is a piece of cake. Some load tests on the system will quickly reveal the limitations.

There is no other confidence like being able to see actual measurement results. Leaving this step to feelings of how scalable your system is, will result in unexpected reality on production. For multiple systems, we found out much different results than we thought we would get of how scalable the system was. This led to finding out the problems we missed in the design, fixing them and releasing a system that truly met the scalability requirements.

The second lesson is, do not wait for later to perform load testing. Load tests should be prepared and performed as early as possible. You will be surprised to find out limitations you did not expect and you can fix them on time.

3. Establish a clear understanding to avoid the fallacies of distributed systems design

So many systems suffer from problems that can be avoided by using solutions to avoid the fallacies of distributed systems design. In the toolset to avoiding these problems we find tactics that effect the architectural design as well as the construction.

The ones that affect the architectural design, such as inverting flow of data to achieve temporal decoupling and using messaging instead of RDP calls, need the understanding of the problems and convincing abilities to have them in place in the system design.

The others that are handled as part of the construction (e.g. making a process idempotent; handling messages in a way to automatically overcome transient faults; making sure the outcome is correct regardless of the order of which the messages get processed by concurrently running processes) can be implemented as reusable libraries.

Conclusions

There are a lot of factors to consider while designing systems that fulfill the desired outcome. While learning more on the subject and developing reusable libraries to take away the repetitive tasks that handle common fallacies, going ahead with actual implementation and early load testing goes a long way to achieve the desired scalability of the system.