Why Your Microservices Architecture Needs Aggregates
By: Dave Taubler
Tags: Microservices, Software Development, Programming, Design Patterns, Domain Driven Design

Why Aggregates?

We’ve looked in-depth at what Aggregates are, and explored ways to identify our Aggregates. Clearly, it takes some up-front effort to design our Aggregates. So, why should we care in the first place?

When defining the pattern in Domain-Driven Design, Evans focuses almost exclusively on the Aggregate as a mechanism for transactional enforcement of invariants.

But this pattern — in which we identify atomic collections of entities with a single externally-accessible reference — becomes useful in many other aspects of our microservices architecture.

In addition to providing that enforcement of invariants, Aggregates help us to avoid later problems caused by things such as:

Unwanted dependencies between entities.

Leaky object references.

Lack of clear boundary around groups of data.

Let’s look at some examples of these issues, and how Aggregates would have helped.

Microservice and data schema design

Let’s take a look at a typical monolithic database. Typically, over the years, we’ll have developed a large database schema, replete with foreign key references throughout.

Starting at any arbitrary table and tracing all of the FK references to and from that table, we’d likely find ourselves traversing the entire schema.

A small but very monolithic database schema

Even with a monolithic codebase, this doesn’t smell quite right.

For example, when making a database call to retrieve an Order , how much data should be returned? Certainly the Order details such as status, ID, and ordered-date.

But should we return all of the Order Items? The addresses where the item was shipped from and to? How about the User objects representing the order-placer and recipient? If so, how much data should come along with those User s?

As we move towards microservices, we’ll be breaking apart our monolithic data schema as we break apart our monolith codebase. This will likely be the most difficult task that we face as we get started.

Fortunately, thinking in terms of Aggregates provides us with a blueprint, and solid guidelines, for designing our data microservices and their associated database schemas.

Rather than arbitrarily drawing lines, and debating which objects “feel like” they belong together, the Aggregate pattern tells us to identify:

Our root entities.

Value objects that would be attached to our root entities.

Invariants that are required to maintain data consistency across related entities.

While it will still take work, and often many iterations, to settle on our Aggregates, we’ll have a guiding light to direct us. And we can be much more confident that we’ve gotten it right, once we’ve formed our Aggregates.

Sharding

Most databases can handle an enormous amount of traffic. But even the most highly-performant database can only handle so much. When we get to the point where our data volume has gotten too much for our database, we have a few options.

One common option, sharding, describes a way to horizontally scale our databases. When sharding our database, we are effectively creating multiple copies of our schema, and dividing our data across those copies.

So, for example, if we create four shards, then each shard would store approximately one-quarter of our data. The schema would be the same across the shards — each one would consist of the same tables, foreign keys, and other constraints, etc.

With sharding, we horizontally scale by splitting a large schema into multiple smaller, identical schemas

Critical to effective sharding is a sharding key. Effectively, a sharding key is a common identifier that is run through a hashing or modulus function to determine which shard it belongs to.

For example, if we’re attempting to update a user, we can take that user’s ID, hash it, and mod it by four (assuming four shards) to determine in which shard we can find the user.

Now, if we imagine a typical monolithic database schema, this might seem like an impossible task.

Why? Well, in our monolithic schema, we will likely have a number of foreign key relationships. For example, we might have a foreign key from the ORDER table to the USER table (to represent the user who placed the order).

Now, we might be able to easily determine where to find a given USER record with an ID of 12345 (12345 % 4 = 1, so that USER record would be found in Shard 1).

But what if a foreign key to that USER record was held by an ORDER record with an ID of 6543 ? 6543 % 4 = 3, so that ORDER record would be found in Shard 3. Given the foreign key relationship, then, this would be impossible to implement.

While this is a clear example from a monolithic database, we could just as easily paint ourselves into a corner with a microservice’s data schema.

Imagine that we’ve created a User service in which — much like our previous examples — a User entity is associated with 0..n email addresses, mailing addresses, and phone numbers.

The underlying data schema would then look like the following:

Now, let’s pretend that we’d eschewed the idea of Aggregates when we built out this microservice. Instead, we’d provided endpoints that allow direct access to all entities, like so:

GET /users/{user-id}

GET /users/phones/{phone-id}

GET /users/emails/{email-id}

GET /users/emails/{email-id}

A year later, our user base has exploded, and we’ve decided to shard. But at this point, can we?

The example below shows our four USER shards, and a sample USER record with an ID of 12345 (12345 % 4 = Shard 1) and associated PHONE_NUMBER record of ID 235 (235 % 4 = Shard 3).

That’s… not going to work

We’ve run into the same problem as with the monolithic data schema.

If we had properly defined our User Aggregate, of course, we would have ensured that every request travels through the root entity. So, it is the root entity’s ID that determines where every entity — including that phone number — belongs.

In our example above, all of the entities — email addresses, mailing addresses, phone numbers, and the root entity itself — associated with user ID 12345 would be stored in Shard 1.

Message passing

Let’s take a brief detour, and mention the bounded context. This is another extremely useful pattern borne out of Domain-Driven Design.

Among other things, it helps us to understand that — rather than a mess of synchronous API calls — our microservices architecture should leverage message passing.

Any time an event occurs within one bounded context, that event will be published to an event bus like Kafka, to be consumed by a service in another bounded context.

Now, the question usually arises: “What should the message contain?” For example, let’s say a User adds a phone number. Once that change is committed to its data store, we want to publish that edit as a message.

But exactly what should we publish? Generally, we want to publish the new state of the modified data. So, we could simply publish the new phone number:

That might be sufficient. Unfortunately, it’s hard to say what additional information the message’s consumers might need. Some consumers, for example, might need to know if the new phone number is also the User ’s primary phone number.

But what if the primary flag is false… and the consumer still needs to know which phone number is the primary?

Hmm. Maybe we should send all of the phone numbers. But… what if another consumer needs to notify the User that the change has been processed, and needs to do it via email? Maybe we should send all of the User ’s email addresses as well?

Clearly, this process might never end… and we might never get it right.

An alternative approach that some teams try is to simply send the ID of the modified entity in the message. Any consumer can (nay, must) then call back to the event publisher to obtain the details of the event.

This approach has two unfortunately problems:

It will, from time to time, result in the wrong data being retrieved. Say entity 123 is modified, and the corresponding message published. Then the same entity is again modified. After that point, a consumer consumes the first event and requests entity 123. That consumer will never pick up that first modification. Now, that might not matter; it could be that the consumer only ever cares about the latest version of the entity. But as producers of the event, we don’t know whether any of our consumers — present and future — might need to track individual changes.

Worse, it turns our nicely decoupled event-driven architecture back into a tightly-coupled system bogged down by synchronous calls across bounded contexts.

So what should we pass as our messages?

As it turns out, if we’ve embraced Aggregates, then we have our clear answer. Anytime an Aggregate is changed, that Aggregate should be passed as the message.

We know this because an Aggregate is an atomic unit. Any change to any part of the Aggregate means that Aggregate as a whole has been modified.

How that Aggregate is represented in the message, of course, depends on our organization. It might be a simple JSON structure, or it might be represented by an Avro schema.

The Aggregate’s data may or may not be encrypted. But regardless of the data format, thinking and designing in terms of Aggregates makes questions like these no-brainers.

Retries

The concept of message passing is often coupled with that of retries. One of the beautiful things about a message-based, event-driven architecture is that resilience — in the form of automatic retries — is essentially baked in.

What do we mean by that? When a message is published to an event bus like Kafka, it’s meant to be consumed by a downstream consumer. Most of the time, things will work smoothly.

Occasionally, however, a consumer will have a problem consuming a message.

Maybe the consumer’s database will temporarily be unavailable, rendering the consumer unable to correctly handle the event on its end.

Or maybe a security appliance is briefly unavailable, preventing the consumer from being able to decrypt the message.

In such cases, the consumer will not move on to the next message until the current message is processed, and the consumer is able to acknowledge the handling of the message. This happens by default with systems such as Kafka. Effectively, the consumer will keep trying until it succeeds.

Often, this is the desired behavior. Usually these problems resolve relatively quickly (although sometimes with the help of an ops team). In the meantime, there is no sense trying to process the next message, as the same problem will likely occur with that message.

But there is a second class of problem: when there is a problem with the message itself. Maybe the message became corrupted in transit. Maybe it contains a bizarre special character (did somebody cut and paste a Microsoft Word file?). Maybe it’s failing some validation check.

In such cases, the consumer could retry the message a million times; it would never succeed.

When detecting such a problem, the consumer might set the current message aside, perhaps into a special triage queue, and continue processing the subsequent messages.

But there’s a problem with this approach. We’ll want to ensure that the “bad” message eventually does get processed, even if it takes some amount of manual effort.

But… what if, in the meantime, the consumer processes a message in which the same data was changed again? That change will be overwritten when the older, “bad” message is finally re-processed.

The diagrams below depict this problem:

Entity 123’s “foo” value is changed to “bar” in Bounded Context 1. So a message representing that change is published; however, it cannot be parsed by the consumer in Bounded Context 2. So it is shunted off to a triage queue.

Sometime later, Entity 123’s “foo” value is changed to “baz” in Bounded Context 1. So another message — representing foo’s change from “bar” to “baz” — is published by the first Bounded Context. This time, the message is consumed by Bounded Context 2, which now sees entity 123’s “foo” value as “baz”.

Some time after that, the initial bad message is fixed (say, a bad character is removed), and it is resent to Bounded Context 2… which now incorrectly sees entity 123’s foo value as “bar”.

This is a problem with ordering. Generally speaking, we need to make sure that events are processed in the order in which they occur. But in the scenario described above, that’s not possible. Or… is it?

If we’ve designed our data around Aggregates, then we know the scope of changes within any given message our consumers might receive. In other words, any message we receive will depict a new version of an Aggregate.

And that Aggregate can be easily identified by its root’s globally-unique identifier (GUID). So, if our consumer determines that a message cannot be processed without manual intervention, it can shunt that message off to a separate queue.

Moreover, it can note the GUID of that shunted message. If it encounters any more messages containing that same aggregate, it can likewise shunt that message off to the same queue.

And it can continue doing that until the original problem is resolved (maybe the consumer needs to be updated to handle that weird Microsoft Word special character). At that point, the “shunted” messages can be fed to the consumer.

It’s certainly not easy to build such a retry mechanism. But with Aggregates, it’s at least possible.

Caching

Caching is another topic that can become unwieldy without well-defined, bounded data structures.

Most caches operate like large hashmaps; they allow us to associate some chunk of data with a single identifier, and to later pass in that identifier to retrieve that chunk.

If we haven’t designed our data around Aggregates, it can become difficult to figure out what type of data we want to cache. Imagine a system that is frequently queried but infrequently modified.

In this system, we might want to cache our query results higher up in our stack to minimize trips to the database. Fine. But what should we cache?

We could simply cache the results of every query. Back to our user example, that means that we could be caching the results of things like:

Searching for a certain user.

Searching for a certain phone number.

Searching for a collection of email addresses.

Searching for the marital status of a given user.

Notice that we’re potentially duplicating data. We’re caching a user object, but we’re also caching individual contact information and groups of contact information, as well as individual fields from the user object.

That has ramifications, of course, in terms of the amount of memory required. It also has more serious ramifications when it comes to cache invalidation.

Imagine that an attribute of a cached phone number changes — from our earlier example, let’s say the “best contact” flag is changed from false to true. So, we invalidate the cached phone number.

But do we also need to invalidate the cached user object? What about the other piece of contact info, that had a corresponding “best contact” change from true to false?

If we’re using Aggregates, we don’t need to worry about these issues. With an Aggregate, we have only a single possible cache key: the Aggregate root’s GUID.

When we retrieve the Aggregate, we cache it. When any attribute of the Aggregate changes, we invalidate the entire Aggregate.

Problem solved.

Service authorization

Well into one of my previous company’s move to microservices, I headed a team tasked with implementing service-to-service, data-level authorization.

In other words, we’d already solved the problem of “is Service A permitted to access Service B?” We needed to solve the problem of “is Service A permitted to request Entity 123 from Service B?”

This meant that we needed to be aware of the current user-agent (for example, the customer who had initiated the request). No problem; that’s what things like JWTs are for. We could pass the user’s ID in a token while making service-to-service calls.

We also needed to be aware of whether that user-agent was permitted to view any particular entity. In our case, the number of potential entities was huge.

In addition, a user might be viewing their own documents, or they might have been given permission by another user to access their documents (for example, by granting power-of-attorney to a third party).

Our goal was to provide a generic, pluggable solution. We also wanted to avoid repeated synchronous calls to a separate service to determine whether a given user had access to a given entity.

For that reason, we decided to determine the items that a given user was permitted to access — once, during startup — and include the IDs of those items in the user’s token.

Had we not designed our microservices around Aggregates, this would not have been feasible. The list of potential entities would have been prohibitive.

However, because we had invested up-front in using Aggregates, we had already constrained ourselves to looking up any entity using its Aggregate’s root’s ID.

Therefore, we only needed to track the Aggregates to which a user-agent was granted access. That list was quite feasible.

Tracking changes

We may find ourselves tasked with tracking changes to our data. Historically, we’d recorded data changes by implementing a change data capture (CDC) system triggered by low-level database activities.

More recently, organizations have tended to move towards capturing changes to business entities, rather than changes to column in databases.

So, we’re faced with a question: “What data should be in the snapshot, and how will we use it later down the line?”

As you might imagine by now, answering these questions will be straightforward if we’ve designed our data around Aggregates. Any time a change to any entity is made, we record the new version of its Aggregate. This is not only simple; it’s also more accurate.

Recall that the original purpose of Aggregates is to transactionally enforce invariants. So, each snapshot of the Aggregate will represent the result of any such transactions.

Retrieving the changes later on also becomes much more straightforward. If we want to see the history of a User’s contact information, we won’t need to worry about gathering changes across multiple CDC tables.

Instead, we just go straight to the Aggregate’s table. Likewise, diffing changes becomes trivial; we simply compare one version of an Aggregate to another.

Myriad others

This was a non-exhaustive list of the challenges that designing our entities around Aggregates helped us solve. Undoubtedly, some of us will find others (try implementing the Command Query Responsibility Segregation (CQRS) pattern without Aggregates!)

When we think about it, it makes sense. Applying the Aggregate pattern forces us to think up-front in a methodical way about which entities belong together.

Ultimately, we’ll have constrained ourselves to entities within well-defined, atomic grouping with a single access point. We won’t wind up with those accidental dependencies between entities, or with the sorts of leaky references that will prevent us from implementing scaling solutions.