Choosing a database and designing a schema for storing an evolving model
By: Duo Yao
Tags: Jsonb, Postgres, Mongodb Atlas, Documentdb, Dynamodb

Choosing a database and designing a schema for storing an evolving model Duo Yao Jul 23·4 min read

Our team came up with a project to build a licensing service. This new microservice needs to store licensing information for modules of our product. Licensing attributes for each module are not well known. The number of modules the platform will support is also unknown.

So here comes the challenges or data persistence requirements:

A database that can accommodate an evolving schema.

A database that supports a flexible query model.

A database or schema which will facilitate migrations to accommodate for potentially shifting models.

A data schema which will easily allow for shifting licensing models or an unknown number of supported models.

With the requirement of supporting flexible schema and our services are hosted in AWS, we decided to opt for a cloud managed NoSQL database. We chose not to go with the typical relational database model mostly to accommodate for the dynamic nature of a licensing model.

There are several database options we considered:

AWS DynamoDB

AWS DocumentDB

MongoDB Atlas

AWS RDS with Postgres using JSONB columns

In the following sections, we list pros and cons for each of the above options and why we ended up going with PostgresSQL JSONB.

AWS DynamoDB

Pros:

Natively supported in AWS. Admin does not need to manually configure infrastructure. DynamoDB automatically scales.

Our company has existing process for purchasing and subscription.

NoSQL key-value store that also supports document data. We can store a flexible-schema document in this database.

Cons:

Weak query model that supports global and secondary index query. This failed our requirement to support more than trivial query patterns.

In summary, DynamoDB performs well in terms of reliability and simplicity. It is relatively straightforward to setup and connect. It is a great fit if your application needs a flexible schema model with support for key-based query patterns. However, queries on non-unique keys or the opportunity to potentially support relatively complex queries is restrictive so it does not meet our query requirements.

AWS DocumentDB

At the point of writing, DocumentDB is the other NoSQL database natively supported in AWS.

Pros:

Our company has existing process for enabling supported AWS managed services.

Claims to be MongoDB compatible, which allows for the potential of running on a private cloud.

Provides better flexibility in terms of query support.

Cons:

A relatively new database added to the AWS family with public release from Jan 2019, means some of its APIs may not be as mature as counterparts.

Documentation claims it to be MongoDB compatible, but from the MongoDB website we found articles which negating this claim.

DocumentDB is a better option than DynamoDB in terms of query support. Additional web search did not reveal sufficient community articles and discussions on this database. In addition, the inconsistencies between Amazon’s Mongo workload support is a cause for concern, disqualifying DocumentDB as our NoSQL choice.

MongoDB Atlas

This database was our first choice for a NoSQL store for the following reasons:

Pros:

Wide public adoption, rich resource and community support.

Rich JSON document model and the ability for potential complex queries within the document structure.

Fully managed NoSQL database by MongoDB.

Cons:

Lack of a straight forward corporate policy for vendor-based subscriptions.

Lack of a straightforward procurement model for AWS hosted Mongo.

MongoDB was our preferred choice. It supports a rich document model. However, the lack of an established corporate process in subscribing to the mongoDB Atlas service eliminated this database choice.

Postgres with JSONB column

The team further researched an alternative and found we can also achieve a flexible and document based schema with a JSONB column from Postgres.

Pros:

Flexible document model with JSONB support.

Already established process for deploying PostgresSQL and related migration scripts.

Cons:

Query on JSONB internal attributes is not as easy as query on standard columns. So potential complex query capability may not be as rich as MongoDB.

Given our specific use case, we can achieve the best of both worlds with PostgresSQL. We formulated a model which would allow for static or fixed license properties and dynamic or flexible module-specific properties.

Our application only needs to query on the fixed license columns such as “status”, “start time” and “duration”. It does not need to query module specific functionality properties as these attributes are only of interest to specific modules imposing constraints based on the internal license structure.

With the above reasoning, we decided to use standard columns to store common license attributes and use JSONB column to store module specific functionality attributes.

Takeaways

We analyzed several database choices for our licensing service and decided to go with a NoSQL option. We further analyzed numerous NoSQL database options and decided to go with PostgresSQL’s JSONB column. Even though PostgresSQL was not our first choice, it ended up being our document of choice due to corporate policies, current team’s knowledge and JSONB’s support towards our primary use-case of flexibility.

Choosing a technology depends on multiple factors, not all of them are technical. We need to consider both the functional and business requirements. When there is a blockage stopping us from going forward, taking a step back to re-evaluate the situation usually results to a solution which falls somewhere in the middle.