Battle of the RDBMS Giants in AWS
By: Vithulan Mv
Tags: Aurora, AWS, Performance, MySQL, Postgres

Why Amazon Aurora?

With the data volume being huge, and the read rate high (about 600 selects/sec at peak), it becomes necessary to have a highly scalable distributed DBMS solution.

According to the CAP theorem, distributed systems can only have two of these three options: Consistency, Availability and Partition Tolerance.

However, it is also known that network failures (partition tolerance) are unavoidable in distributed systems.

CAP theorem

In our use-case, we needed to meet these two criteria:

The system should always be available. The data should be distributed across nodes for high availability and durability.

This makes our system an AP System (Availability-Partition tolerance).

AWS provides RDS and Aurora when it comes to RDBMS solutions. As per official AWS documentation, they provide 99.99% availability, replicating 6 copies of the data across 3 Availability Zones and backing up the data continuously to Amazon S3 with the Amazon Aurora solution. There may be an inconsistent state during replication or communication failures between nodes. However, Amazon Aurora achieves near real-time consistency within milliseconds.

They also claim that Aurora MySQL is five times faster than RDS MySQL, and Aurora PostgreSQL is three times faster than the RDS PostgreSQL.

Therefore, given the facts, it was a no brainer to choose Amazon Aurora.

The next step would be to decide on which Aurora DB to choose. Given the importance of this, we conducted a few experiments with MySQL Aurora and PostgreSQL Aurora to help us decide which of these would be the best fit for our use case.

Optimizations

We found the best solution with both MySQL and PostgreSQL after conducting shardings at the database level by creating different databases to store similar data which came from the same source, and conducting multiple optimizations to the query and tables by experimenting with different indices.

Sharding is achieved through an ETL (Extract Transform Load) process. It decides from which database to load the incoming data, and the system finds the relevant database to retrieve the data from based on the user request.

The MySQL table schema and the query are shown below (after masking the attribute names and table names) to give you visibility before you proceed into the performance test results. Similar schemas and queries were developed and used in PostgreSQL with default settings.

MySQL Table Schema:

CREATE TABLE IF NOT EXISTS `table_1` (

`att1` VARCHAR(9) NOT NULL,

`att2` TINYINT(1) UNSIGNED NOT NULL,

`att3` VARCHAR(14) NOT NULL,

`att4` DATETIME NOT NULL,

`att5` BIGINT(10) UNSIGNED NOT NULL,

`att6` BIGINT(10) UNSIGNED NOT NULL,

PRIMARY KEY (`att3`, `att1`)

) ENGINE=INNODB DEFAULT CHARSET=UTF8; CREATE TABLE IF NOT EXISTS `table_2` (

`att1` VARCHAR(9) NOT NULL,

`att2` TINYINT(1) UNSIGNED NOT NULL,

`att3` DOUBLE(13 , 3) NOT NULL,

`att4` DATETIME NOT NULL,

`att5` BIGINT(10) UNSIGNED NOT NULL,

`att6` CHAR(1) NOT NULL,

`att7` BIGINT(10) UNSIGNED NOT NULL,

`att8` BIGINT(10) UNSIGNED NOT NULL,

PRIMARY KEY (`att1`,`att2`,`att4`, `att5`)

) ENGINE=INNODB DEFAULT CHARSET=UTF8

The MySQL query that was used (attributes and table names are masked):