The story of a micro-service transformation in Hepsiburada
By: Cem Başaranoğlu
Tags: Ddd, Microservices, Ecommerce, Hepsiburada, Transformation

We launched the new checkout ecosystem “Erebor” after a few months ago in Hepsiburada. In this article, I will talk about what we have experienced during in this micro-service transformation.

“A software ecosystem is the interaction of a set of actors on top of a common technological platform that results in a number of software solutions or services.” Software Ecosystem: Understanding an Indispensable Technology and Industry by David G. Messerschmitt and Clemens Szyperski

I prefer to use “ecosystem” as a keyword to describe such a large structures because of their depth, capabilities, and scope. This is exactly my purpose of using this term in a few places in the rest of the article.

strong together

We work as vertical product teams in Hepsiburada. Obviously, all of these product teams are working autonomously as an independent startups, and together with all these teams we are creating Hepsiburada. The Checkout team is just one of thirty product teams in Hepsiburada. We try to make our customers’ lives easier by managing the products and technologies behind the process which is starting from the basket flow until completing their orders. We divided our monolith checkout system into nearly 40 micro-services with the Erebor. I will try to share with you what we learned in this journey in three chapter.

CHAPTER ONE: DEFINITION OF THE PROBLEM

CHAPTER TWO: CHOOSING THE RIGHT TECHNOLOGY SPECS

CHAPTER THREE: THE COMPLEXITY BEHIND THE SIMPLICITY

CHAPTER ONE: DEFINITION OF THE PROBLEM

In fact, the first step of all good solutions starts with defining the problem well.The “ultimate” correctness of the solution depends on the exact definition of the problem.We gathered our problems under the four different topic based on this fact.

There is no bridge between the problem space and the solution space

Different loads in same spot

Survive in dependency hell

Stuck at the borders

We improved our entire ecosystem with the solutions of these topics.

There is no bridge between the problem space and the solution space

problem and solution spaces

The problem space contains all stakeholders of a product. These stakeholders have comprehensive knowledge of their products. They can produce many non-technical solutions related to the their products. On the other hand, the solution space is a space in which engineers are involved in different roles and produce technical solutions. The productivity of our product and the quality of our code-base have been adversely affected over time because of this complex communication problem. Also, the complexity has become the standard for us rather than an option against simplicity.(unfortunately)

DDD is a bridge between problem space and solution space

So, We used “DDD” to solve this complex communication problem. In fact, we could preserve our code-base for a certain period of time by implementing different refactoring methodologies or an extreme programming paradigms. All resistance mechanisms will lose their efficiency over a time as long as there is no common language between these two different spaces. Actually all resources about DDD lead us to the following truth.

DDD builds a solid bridge between these two spaces with using the ubiquitous language.

There are many ways to design the ubiquitous language. We chose “the domain storytelling method” invented by Stefan Hofer and Henning Schwentner because it uses imagination and stories to simplify even very complex problems.

The best way to learn a language is to listen to other people speak that language. Try to repeat what you hear and mind their feedback. Gradually, you will progress from individual words to phrases and to complete sentences. The more you speak, the faster you will learn.

We established four golden rules to keep it all on track when applying DDD.

Learn the business first and than design the domain.

the business and the domain. Write or apply your code like you speak.

or your code Don’t worry about the Tactical design , focus on strategy design first.

the , first. Use fitness functions as much as possible.

Tools of DDD

We learned that mistakes made in tactical design could be resolved quickly. However, the mistakes made in the strategic design unfortunately caused the anemia of the domain.(anemic model). Also, We have implemented as many fitness functions as possible in order to keep the system robust. We are currently trying to integrate the concept of “Fitness function-driven development” into our system.

As a result of all this, we divided our system into four main domains.

Basket

Delivery

Payment

Snapshot

The Domain Design (sample)

First of all, we designed our ubiquitous language. Then we determine our bounded-contexts for each domain.

After deciding on our bounded contexts, we modeled our communication standard with these contexts via context mapping tools.

After strategic design, we started to use tactical design tools. We have defined our aggregates, entities, value Objects, repositories, factories and services.

Different loads in same spot

Our Monolith checkout service was used by about 10 different teams.At this point, all availability problems in our service were directly affecting these teams. The metrics we have collected here have led us to the following conclusions.

We have large difference between the number of reads and writes. We must scale independently both sides.(10/3)

The performance is critical. We can optimize read and write sides independently. Also, we support a low of parallel operations on the same set of data.

We must normalized the write database .We must make writes efficient but We don’t need the normalized data in read side (projection-per-client/projection-per-business)

Based on these data, we decided to implement the CQRS with Event Sourcing. As you know, There are three types of CQRS.

Separated class structure using domain classes for commands and DTOs for returning read data, which will introduce some duplication.

using domain classes for commands and DTOs for returning read data, which will introduce some duplication. Separated model with different APIs and models for reads and write respectively. In addition to optimized queries this also enables caching, making it interesting for high load on reads. )

and models for reads and write respectively. In addition to optimized queries this also enables caching, making it interesting for high load on reads. ) Separated storage optimized for queries enabling even more scaling of reads and separate types of storage for writing and querying, e.g. a relational database and a NoSQL type. Synchronization of read storage commonly runs in the background causing eventual consistency on the read side. Together with the best scalability this pattern also brings the highest complexity.

Separated storage

We chose the the separate storage strategy because it solves our problems more clearly than others. But you know, ‘Everything in life is a trade-off.’

We recommend that you research the CQRS myths before applying the CQRS. But it’s worth sharing what Greg Young shared on stackoverflow.

We had to find answers to some basic questions while implementing the CQRS with Event Sourcing.

Did we need to apply to all domains, really?

No. In fact, this solution fully defines “The Complexity Behind The Simplicity”. If we applied this to all domains, we would have lost our main motivation, “simplification”. Also, it will not be sustainable for all domains due to its cost. We have decided to implement this solution only in the “basket”, which takes the most load and needs this solution the most. However, we applied CQRS with the Separated model with different APIs in some domains considering the cost, feasibility and sustainability.

How were we going to prevent losses that could occur in events?

The processing of data without any delay is a very critical issue for us.Our system is definitely not suitable for an eventually-consist solutions.We could use Microsoft Distributed Transaction Coordinator or Transactional Outbox Pattern to prevent losing an event. Also we could distribute the load in a balanced and consistent way with consistency hashing. However, we did not choose them because they caused minor delays. We use 2pc to protect the integrity of an events.

How would we ensure data consistency on the read and write side?

In fact, we obeyed Greg Young on this step. We used hash codes to achieve the above mathematical function. In other words, We have ensured that the hash codes of the projections we created by combining our events are the same as the aggregate. Here, we have created different solutions to avoid some exceptional situations as follows.

— We have stored the summary of events as metadata to ensure data integrity. In this metadata, we store the number of events on aggregate and the hash code of aggregate.

— We re-build the events in case the projection data is not available.We check the data consistency and reliability of new projections by comparison with metadata.

— If the data is still unreliable, we rebuild an events using an aggregate.Thus, we destroy all existing events and build reliable projections again.

So what did we get at the end?

We solved many availability problems in our entire system with CQRS and Event Sourcing. In the basket domain where we use these two solutions;

We can process 2.5M events at approximately the same time in 60MS.

We have a projection-store hit ratio of 98%.