Query Service using CQRS and Aggregator Pattern
By: Ahmad Hafizh
Tags: Cqrs, Software Architecture, Technology, Microservices, Microservice Architecture

Persistence microservice is common while building a microservice. In this microservice persistence, microservice has its own database. The CRUD (Create, Read, Update, Delete) process is also carried out on this microservice’s persistence in its own database. For example, Job Service, with a connection to this own database, wants to get data on this Job Service to make an API request for this Job Service.

More data added might slow down the CRUD process.

If there is more data on this Job Service, the CRUD process in this Job Service will be disrupted and begin to slow down. We’ll start thinking about how to do scaling this service. One of the things we usually do is speed up the reading process by indexing specific fields. By indexing this particular field, we can speed up the service’s read process, but we will sacrifice our write process. The writing process will be slower due to indexing.

Representing Read/Write Process Using CQRS Pattern

To solve this problem, we can approach using the CQRS Pattern. CQRS itself stands for Command Query Responsibility Segregation, where we will separate the write (Command) and read (Query) processes on different services.

For example, Job Service has a MySQL database. We will separate the Command and Query processes in this Job Service. For the Command process, namely Create, Update, and Delete, we will do the Job Service. As for the Query process, Get, we will do it on a new service, namely the Query Service, which has its own database, for example, Elasticsearch.

Applying CQRS into two services is expected to speed up the process.

By separating into two services, the Command and Query processes become more visible. One service only handles writes where the database is not indexed on the fields. One more service can handle the reading process where the database can index or analyze the field. But the question is how to synchronize the MySQL database owned by Job Service with the Elasticsearch database owned by the Query Service.

The way that can be done is when the Job Service will write data to the database, the Job Service will also make an API request to the Query Service to store data in elasticsearch. However, this method is not quite right. Because the process that occurs will delay, increase latency, and also when the Query Service is down, an error will happen in the Job Service write process. Even though the problem is from the Query Service and it could be that the Job Service is still safe.

CQRS process of Job Service and Query Process.

We can make an approach using a Message Broker, for example, Kafka. So when a Job Service does a write process, the Job Service will produce a message event to Kafka. Then Query Service will consume message events on Kafka to save data on elasticsearch. This is more recommended because if the Query Service is down, it will not interfere with the Job Service writing process, and the data is still stored in Kafka. So when the Query Service is up again, the Query Service will consume message events that were left behind during the downtime.

Using Kafka as a tool for the CQRS process.

With this mechanism, we can apply the CQRS pattern to our microservices. However, I had encountered a problem where when I made a read request for Job data, it turned out that it still needed data from other services. For example, from Agent Service and Gamification Service. So it takes aggregate data for Agent data and Gamification data.

Aggregator is a facilitator to accommodate data from other services

One thing that can be done is to create a service aggregator. Where the aggregator service accommodates the aggregate data needed by other services. For example, here, we will change the Query Service as an aggregator. So the Query Service will consume the writing process of several services, namely Agent Service and Gamification Service. So when the client makes a read request on the Query Service with a request that requires data from the Agent Service and Gamification Service, the data is accommodated in the Query Service database, which is an aggregator of other services.

Response time is speeding up while BIG Agent Team uses the CQRS.

BIG Agent team has already used the CQRS pattern for about 3 months. The result is quite good because it can reduce response time by 50%. Next, we will continue to use and we will try other services.