Rethinking the Orchestrator
By: Gonçalo Garcia
Tags: Distributed Systems, Backend Development, Software Architecture, Software Development

(Unknown image author)

It was a rare rainy August morning in Lisbon as the first developers of the Routing team arrived at a quiet office. Like every other day, the QA Specialist greeted them as they arrived, but today something felt off. There was a slight uneasiness in the air, a hint of apprehension intensified by the gloomy weather. They did what every developer would do in the situation. They opened Jira. And what they saw threw their world into an immediate stall. The Orchestrator had emitted the same event twice. War-torn and battle-scarred they knew what it meant. They knew right away it was a race-condition and thus they became silent. For hours they were silent, as silent as anyone had ever been. And in silence they remained, while the patter of the rain and the mechanical keyboards echoed around them.

At least that’s how I imagine it (I wasn’t there at the time).

The reality is that, six months later, many other instances of race-conditions had been found in the product which meant that a critical flaw in the design needed to be addressed.

In this article, we will guide you through the process of rethinking and redesigning our system to be thread-safe.

What Is the Orchestrator?

To put it in simple terms, the Orchestrator is a distributed state machine that supports concurrency. It is used as a runtime for complex functionality that is built in the form of Routing Configurations — directed graphs (in JSON format) that describe a set of steps to be executed. These steps can be computational (i.e. starts, forks, joins, decisions, terminations), or function-steps, which are small programs built (by internal teams) to perform custom operations.

The Orchestrator’s architecture is comprised of four components:

Figure 1 — The Orchestrator’s architecture.

Interactions REST API — used to trigger a start step in a configuration, creating a new flow.

— used to trigger a start step in a configuration, creating a new flow. Message Queue — a RabbitMQ cluster that queues configuration steps for execution.

— a RabbitMQ cluster that queues configuration steps for execution. Engine — this component will consume steps from the queue, evaluate them, and then enqueue an event that will signal the runtime to proceed to the next step. This helps us increase performance by scaling the number of engines and guarantees that no interactions will remain unfinished due to a failing engine.

— this component will consume steps from the queue, evaluate them, and then enqueue an event that will signal the runtime to proceed to the next step. This helps us increase performance by scaling the number of engines and guarantees that no interactions will remain unfinished due to a failing engine. Data Store — the Engine is completely stateless to improve scalability and fault tolerance, so all of the operational data is stored in a MongoDB cluster.

What’s Wrong with It?

The first issue we discovered was related to a configuration that contained a fork step, creating two flows that would eventually reach two separate termination steps.

When steps are executed, they can trigger events that are listened to by other systems in order to begin performing certain computations.

One of those events is triggered by the termination step after all other flows have reached their termination and the interaction is marked as finished. This event tells other systems to begin their post-interaction work.

When the termination step is executed and all other flows have finished, it will check if the interaction has been marked as finished by another flow, and if it hasn’t, it marks the interaction itself. This process is illustrated below.

Figure 2 — The Orchestrator’s Race Condition.

With the configuration described above, we noticed that in certain situations two flows would terminate at the same time, and, since both would see that all flows were terminated, both would check the state of the interaction.

However, due to a race condition, both flows would see that the interaction was still running and would mark it as terminated, publishing the event in the process. This is worrisome because the post-interaction work may not be idempotent.

This sounds like an easy fix, but it was only a symptom of a much larger problem. The system had no structured way to deal with concurrency, and the addition of more complex features that relied on it began consistently breaking two of our invariants:

Events must be delivered exactly once.

All flows must begin with a start step and eventually reach a termination step.

The issue needed to be tackled, and so we set out to find a solution.

Proposed Solution

Our solution was conceptually simple.

We decided to identify all the primitive orchestration operations, such as those that modify the internal state shared between nodes. These operations must be atomic as they are used, for instance, to emit events, transition between steps, and other more complex features. These primitives would then be consolidated under a single interface in the Engine, which would be implemented with thread safety in mind. All features would then have to go through this implementation to execute any of the primitive operations.

We knew that it would be difficult to achieve as the primitives were being used completely ad-hoc, and scattered throughout the codebase.

Getting it Started

Now that the problem had been identified and a solution had been found, we were ready to start working, but first, some important decisions needed to be made.

Rewrite or Fix?

Most of us have felt the urge to completely rewrite our projects at some point in our lives.

As the customer base grows, requirements change, and projects become more complex.

Changing core components is no longer something that can be done in a few days’ work, and you’re absolutely sure you could write a leaner, meaner, cleaner system if your manager would just let you start from scratch.

However, from a management standpoint, this isn’t always easy to justify. Rewriting might take more time, cost more, and a better system for developers might not have clear quality-of-life improvements for the customers.

In our case, the primitive operations were scattered throughout the code. In theory, this could be fixed with a lot of effort but it would result in a hackish solution and a rapid increase in tech-debt.

Additionally, several new developers (including me) had joined very recently, and a lot of them (also me) were quite inexperienced. This meant that it would take quite a lot of time, possibly months, for the team to be fully accustomed to the existing code be and ready for these massive changes.

Another important factor is that our system is not yet being used in production for clients, but lots of teams were already building on top of it for a joint release that was approaching. So, while we had to maintain the whole system running for the other teams in order to meet our deadline, there was less risk for the company.

In the end, rewriting the Engine was a natural decision for us. The newcomers would be able to contribute much sooner, providing a point-of-view that hadn’t been tainted by the previous implementation. The more inexperienced members would have a rare chance to contribute to a large project from its inception, learning from the architectural decisions taken along the way. And finally, the original team would be able to bury the demons in their past.

Feature Flags or Feature Branch

The decision to build alongside the current product had been made, but we couldn’t just create a new module and work on it for a few months while our prescient programmer vision guided us towards the right path (turns out that’s not what Extreme Programming means).

This new module would need to interface with the rest of the project so that it could be functionally tested.

The decision then fell on whether to use feature flags or a feature branch to isolate the new developments while V1 was still being actively maintained.

In day-to-day development we use feature branches to easily parallelize our work, so creating a feature branch for this new endeavor would seem like a natural extension of the process. However, using a feature branch for long term development tends to delay the integration process leading to complex, error-prone conflict resolution, and possibly unexpected incompatibilities.

We are also very accustomed to feature flags in our daily work and they’re usually used to build and roll-out new features progressively. This avoids some of the drawbacks of feature-branches but, depending on the project, may require significant effort to cover all possible entry points. Additionally, it would increase our build times noticeably with the addition of a duplicated module and all of its tests. There’s also the remote possibility that the new guy (me again) will accidentally roll-out the untested Engine in production.

For this endeavor we settled on Feature Flags, the Orchestrator had a very clear entry point meaning we could use a single if statement to switch between both versions of the Engine. With a single flag, we were able to continuously deploy the new changes and it allowed our QA team to easily test both engines in the same environment, which proved to be useful countless times.

Defining the API

As previously stated, we set out to identify the primitive operations of the system and consolidate them under a single API.

The advantage of this approach was that we could start by building and testing these primitives without the overhead of all the higher-level features. Additionally, it allowed us to parallelize our work, and implement multiple primitives simultaneously.

The key here was the test battery that we created to validate the API. There were multiple tests for each primitive, and all of them exploited concurrency in an attempt to simulate race conditions.

As we expected, most of the tests failed, as the API was not yet built for concurrency.

This test-driven approach not only validated our understanding of the existent issues, but it created a baseline for the implementation of structured concurrency.

Getting it Done

Now that the foundation had been built we could start working on a better Engine.

Lock or Not to Lock

The problem of lock vs lock-free algorithms has been widely studied, and thanks to that, we common mortals can now make this decision after reading a single StackOverflow answer. Fortunately, not everyone in our team is a common mortal and we decided to thoroughly test the two approaches before committing to one.

To do so, we built two proofs-of-concept, that took the API we had implemented, and improved it to solve the concurrency issues. The lock-free version would rely on MongoDB’s atomic operations, while the other would be built around a custom-built distributed lock service.

Both approaches can be easily compared by looking at how they solve the problem depicted in Figure 2. Since the crux of the issue is the fact that the MongoDB read and write operations aren’t performed as part of an atomic transaction, solving this with the database’s atomic findAndModify operation becomes trivial, as can be seen below (Figure 3).

Figure 3 — Solving the race condition with atomic database operations.

It’s not possible for Consumer 2 to view and modify the old state while it’s being modified by Consumer 1 by reading and writing in a single atomic step, thus solving the race condition.

The distributed-lock based approach, on the other hand, is slightly more complex. It required creating a MongoDB collection in which each document represents the lock-state of a resource (i.e., locked or unlocked).

When operating over any privileged resource, the consumer must acquire the lock over it, execute its operation, and release the lock.

In the meantime, the other consumers will not be able to obtain the lock and must retry the operation until they are successful (or they exceed the number of retries).

To ensure that there are no race conditions when obtaining the lock, we once again used MongoDB’s findAndModify operation. This approach is depicted below (Figure 4).

Figure 4 — Solving the race-condition with a distributed lock implementation.

It’s clear that the first approach is much simpler, and more efficient, while the second approach is more flexible. MongoDB’s findAndModify operation can only be executed on a single collection, so it cannot be used to solve every problem.

Initially, we hoped that a lock-free approach was sufficient, as locks may degrade performance and one must be careful to avoid deadlocks.

However, tests showed that even after reworking our data model, our operations could not be implemented with the atomicity guarantees provided by the version of MongoDB we license, as many operations required changing multiple collections in a single atomic step. Therefore, our concurrency was solved by distributed locks.

By luck of the draw, both PoCs were implemented by two of the new members of the team. This goes to show the importance of our early decisions, as the newcomers were able to contribute to core design decisions and evaluation efforts right away.

Supporting the Current Function Ecosystem

As we’ve stated before, Routing Configurations can be composed of function steps.

Now, these functions are not made of completely arbitrary code, and a disgruntled developer cannot build a function that lets him play online games instead of working (although we did try in the last Hackathon).

These functions make use of a set of functionality provided by the Engine that we call StepSDK. The SDK allows functions to make remote requests to other internal systems, emit events (not to be confused with the Engine’s events) among other things.

Thankfully the functions only depend on the StepSDK’s API and not its implementation, but the implementation itself was tightly coupled with the Engine V1, especially its data model.

To fix this, we began by first isolating the minimum set of data needed to implement the SDK, while attempting to keep it Engine-agnostic by not relying on the internal data model, which shouldn’t be exposed. Once we did that, we extracted the SDK method implementations to a separate module that only depended on this data (that we called Engine Commons).

We were able to support most of the StepSDK’s features using this approach, with a small subset having to be reimplemented for the Engine V2. Still, this saved us a lot of work, and after this effort, all the functions that already existed for V1 (a number that grew every day) could be used in V2.

This took some effort but it was a major stride towards the roll-out and adoption of the Engine V2.

Learning from Current Mistakes

As we were building the new Engine, version 1 was still being used in production and integrated with other systems.

Often, we would have requests from internal clients to fix or improve some features in the existing product. Usually, these were things that we had previously identified, so they were either small easy fixes or larger changes that we had decided to solve in the reimplementation. “That’ll be solved in V2” became our team’s catchphrase.

However, our assumptions weren’t always right. In certain instances, while the feature we had created would be able to solve the customer’s problems, its semantics did not match what they naturally expected. So we were left with a dilemma, do we teach them how to use the features, or do we pivot and implement it in the way they expected?

The answer to this question was to speak to more users. When consulting other clients we came to the realization that most of them followed the same thought process.

Desire paths prove that it is very difficult to get people to follow a procedure when there are other ways that are easier or better for them.

When faced with the decision, we decided to model our approach around the client’s assumptions, rather than force a different approach upon them and risk having no adoption of the feature.