Data in the era of microservices
By: Steve Meckman
Tags: Open Source, NoSQL, Data, Microservices

In the past ten years, microservices architecture has emerged as a popular method to use to build efficient, responsive, and scalable applications. A microservice runs independently from the other services that make up an application, and a fundamental principle of a microservice is that it is responsible for its own persistence layer, which must be independent of the persistence layers used by other services.

Persistence Layer Requirements

A persistence layer designed uniquely for the service that uses it should not have other dependents that could require a compromise in its schema, nor would it have any uncontrolled inputs that could weaken its data integrity. Any outage, planned or unplanned, would have less of an effect on the entire system if it only affected a single service; likewise, interruptions from any upgrades or maintenance done to it would be minimal.

In an environment of automated CI/CD pipelines and scalable container systems, the persistence layer platform needs to be lightweight and easy to set up and start. Microservice environments are more prone to scale horizontally rather than vertically and need data containers that are able to keep up. This means that the database needs to have self-managed sharding and clustering capabilities.

Data queries are not always simple, but they are expected to be quick. Different types of persistence platforms have their own strategies to store and access data, with benefits and drawbacks depending on use cases. For example, the numerous indices that help make complex queries quick in certain database systems can make it difficult to scale horizontally, a scaling strategy that benefits from the loose coupling of all associated elements.

Search Engines as Persistence

Most popular database systems fall into one of two main categories: RDBMS, which stores their data in more or less normalized tables, and NoSQL databases, which typically store data as a document. Typically they attempt to minimize physical storage requirements, as traditionally, data storage (and by association, redundant data) was expensive. Storage prices have been falling, however. For a very long time (http://jcmit.net/diskprice.htm ). With the cost of a megabyte being a tiny fraction of what it was 20 years ago, maybe this philosophy of attempting to minimize redundant data should be reconsidered.

Lucene-based search engines like Elasticsearch or MongoDB Atlas Search store documents, much like a NoSQL database, but they will store a new document for every small permutation of the data. They focus on finding a very specific data item very quickly, even if the means to do so promotes data redundancy. Typically, this results in extremely large data sets, but in a microservice environment where each service ideally focuses on a single concern, the data schema often is very simple, and unchecked database growth can be mitigated.

Search Engines has all the ingredients of a successful microservice database. It scales horizontally extremely well: it can split an index into multiple shards, each one its own unique index that is searched in parallel with the other shards during a query; its nodes can be clustered; properly optimized, it has an incredibly fast response time.

Optimization of a query on a search engine can happen at the query level, where the result set can be limited to the number of documents with a response accuracy higher than an established threshold. As another example, the query can also be trimmed to return only the required fields out of a whole document. Optimization can occur at the persistence layer level as well. A properly designed index stores only the data that a service needs, updating itself as needed. While Lucene is extremely responsive, any data repository will return data faster if there is a smaller dataset to search through. Where document-based systems differ from the RBDMS paradigm is that the latter builds its response documents on-the-fly from one or more highly normalized data collections, whereas the former will most often have the response document already fully-formed.

A microindex is a Lucene search index, specifically tailored to the data needs of a microservice. At Advance Auto Parts, we have developed a tool that allows for the granular movement of big data. The microindex, whose creation we will demonstrate on this forum in a subsequent article, is just one of its many use cases.

A service developer, for example, often creates a new system that requires data. The database they create needs to be hydrated somehow. Hydrating a data store, especially in a high-performance, containerized environment, can pose challenges. Oftentimes, an ETL tool will be used, but they often suffer from some drawbacks:

Learning curve (and lack of resources available on the market).

Limited extensibility and proprietary extensions, with potential vendor lock-down.

Scalability issues, as they rely on proprietary technologies and licenses.

Limited inclusion in an automated CI/CD process.

Architecture proliferation and potential issues with corporate compliance.

We have developed a tool that has addressed these issues, by examining and addressing our needs:

Ability to move data from multiple sources and source types to multiple destinations.

Easily executable without a need to call IT.

Fully compliant with our security model.

Fully reusable.

Usable for any type of service we need to build.

This tool addresses the needs listed above and does so in a way that is easily configurable from a single recipe file. We will be introducing this new tool, which we have named Data in the Fast Lane, and we will be expanding on the search use case, very soon.