Comparing Distributed SQL Performance — Yugabyte DB vs. Amazon Aurora PostgreSQL vs. CockroachDB
By: Karthik Ranganathan
Tags: Scalability, Postgres, Database, Sql, Kubernetes

Update: A new post “The Effect of Isolation Levels on Distributed SQL Performance Benchmarking” includes performance results from running these workloads at serializable isolation level in YugabyteDB.

We are excited to announce the general availability of Yugabyte 2.0 this week! One of the flagship features of the release was the production readiness of the PostgreSQL-compatible Yugabyte SQL (YSQL) API. In this blog post, we will look at the performance and scalability of YSQL as compared to two other PostgreSQL-compatible distributed SQL databases — Amazon Aurora PostgreSQL and CockroachDB.

SQL benchmarks demonstrated that YSQL is 10x more scalable than the maximum throughput possible with Amazon Aurora. Additionally, for a similar hardware profile, YSQL delivered nearly 2x the throughput of Amazon Aurora at half the latency.

In case you are wondering what distributed SQL is — it brings together the SQL language and transactional capabilities of an RDBMS and the cloud-native capabilities such as high availability, scalability, fault tolerance and geo-distribution that are typical to NoSQL databases.

Benchmark Setup

The table below summarizes the design points of these databases. Note that we are explicitly not considering multi-master setup in Aurora PostgreSQL because it compromises data consistency.

In this post, we look at the following performance and scalability aspects of these databases:

Write performance

Scaling writes

Scaling reads

Scaling connections

Distributed transactions

All the benchmarks below were performed in the Oregon region of AWS cloud. The benchmark application can be found here. Yugabyte DB 2.0 was setup on a three node cluster of type i3.4xlarge (16 vCPUs on each node) in a multi-az deployment. This deployment is shown as a UI screenshot below.

CockroachDB (version 19.1.4) had an identical setup to Yugabyte DB. Aurora PostgreSQL was setup on 2 nodes of type db.r5.4xlarge (16 vCPUs on each node). One node was the master, the other node a standby for fast failover in a different AZ. This setup shown below.

[Edit] Note that CockroachDB only supports Serializable isolation, while Yugabyte DB and Amazon Aurora support both Serializable and Snapshot isolation. Amazon Aurora even supports the lower isolation level of Read Committed which is also its default. The benchmarks in this post use the default settings in all DBs — which is sufficient for the correctness of these workloads (simple inserts and secondary indexes). The SQLInserts and SQLSecondaryIndex workloads of this benchmark client were used for these results.

Write Performance

The benchmark was to insert 50M unique key-values into the database using prepare-bind INSERT statements with 256 writer threads running in parallel. There were no reads against the database during this period. The benchmark results are shown below.

While the above numbers are already impressive, we are just getting started with the performance of YSQL. YugaByte DB’s core storage engine, DocDB, which powers both YSQL & YCQL, is capable of much higher throughput. The semi-relational YCQL API, which runs on top of DocDB similar to YSQL, is more mature and hence performs better as shown below.

There are additional improvements that we are working on in the YSQL query layer to achieve even better performance (to match that of YCQL).

Scaling writes

What happens when we need to scale? We had noted in the table above that AWS Aurora cannot horizontally scale writes. The only way to scale writes in Aurora is vertical scaling, meaning the node has to be made beefier. The maximum write IOPS Aurora can scale to is limited by the largest available node in terms of vCPUs.

Beyond 1 Million Writes/Sec with Yugabyte DB

Since Yugabyte DB is both high-performance and horizontally scalable, an experiment to scale it to a million write ops/sec was in order. The setup was a 100 node Yugabyte DB cluster with c5.4xlargeinstances (16 vCPUs and 1TB of storage per node) in a single zone. This cluster, named MillionOps, is shown below.

This cluster was able to perform 1.26 million writes/sec at 1.7ms latency!

168K Writes/Sec Ceiling of Aurora PostgreSQL

The above benchmark numbers (28K writes/sec) was observed on a 16 vCPU (db.r5.4xlarge) machine. The largest instance available for Aurora has 96 vCPUs (db.r5.24xlarge), which has 6x more resources than the one used for the benchmark shown above with 16 vCPUs (db.r5.4xlarge). Assuming that the writes scale with the machine size, the maximum write throughput of an Aurora database with multiple tables is capped at 168K ops/sec. Even though Amazon Aurora can store up to 64TB of data, this throughput bottleneck will pose a practical challenge in exploiting the available storage. After this write throughput ceiling, the only choice is to manually shard the data at the application layer, which is a complex undertaking.

By contrast, a Yugabyte DB cluster scales linearly with the number of nodes. A 12 node cluster of Yugabyte DB would be able to exceed the above write throughput of 168K ops/sec. A graph comparing the write scalability of these two databases is shown below.

Scaling Reads

Both these databases can scale reads, however:

Read scaling in Aurora compromises consistency by serving stale reads.

Application design could get more complex if they have to query read replicas in Aurora.

Let us look at how read scalability is achieved in these databases.

The Aurora PostgreSQL documentation outlines the following in order to scale the database.

We have already looked at the write throughput ceiling as a result of instance scaling. Let us examine read scaling in Aurora. Reads and writes have separate endpoints in Aurora. In order to scale reads, it is the responsibility of the application to explicitly read from the multiple read endpoints.

Firstly, this means that applications are required to explicitly include which endpoint to connect to in their design. This decreases the velocity of developing applications because which the endpoint to connect to becomes a part of the application architecture and may not be a trivial exercise, especially when considering failover scenarios.

Secondly, and more importantly, the bigger issue is that reading from a replica returns stale data — which could compromise consistency. In order to read the source of truth, the application has to read from the master node (which also handles all the writes). Because a single node needs to serve consistent reads, this architecture would limit the read throughput to whatever can be served by the largest node (similar to the analysis we did with writes).

By contrast, Yugabyte DB treats all nodes in an identical manner. This improves things in the following ways:

The application simply needs to connect to a random node in the cluster, and the rest is handled by the database. All the nodes of the database can be put behind just one load-balancer.

When performing reads, all nodes of the cluster are able to participate and hence the read throughput is much higher.

Eliminating the Load Balancer with Cluster-aware JDBC Drivers

As an attempt to simplify things even further, we’re working on a cluster-aware version of the standard JDBC driver, called Yugabyte JDBC. These drivers can connect to any one node of the cluster and “discover” all the other nodes from the cluster membership that is automatically maintained by Yugabyte DB.

Events such as node additions, removals and failures are asynchronously pushed to these client drivers, resulting in the applications staying up-to-date when it comes to the cluster membership. With cluster-aware JDBC drivers, you no longer need to update the list of nodes behind the load-balancer manually or managing the lifecycle of the load-balancer, making the infrastructure much, much simpler and agile.

Scaling Connections

Scaling the number of connections is a common concern with PostgreSQL. There is a limit to the number of connections to an Aurora PostgreSQL database. From the AWS documentation, the table below summarizes the recommended number of connections to the database based on the instance sizes.

The table shows that the maximum number of connections recommended, even in the case of the largest Aurora PostgreSQL database, is 5000 (though the theoretical maximum mentioned in the docs is 262,142). In cloud-native applications which have many microservices and massive scale, this quickly becomes a limitation.

With Yugabyte DB, the number of connections is specified per node in the cluster. The default number of connections per node (also configurable) is 300, in our example setup of 3 nodes we would get a maximum of 900 connections. But scaling for connections is easy. By choosing 6 instances with 8 vCPUs (instead of 3 instances with 16 vCPUs), we have effectively doubled the number of connections to 1.8K while keeping the resources the same! Similarly, by choosing 24 instances with 8 vCPUs (rough equivalent of the largest Aurora cluster with 96 vCPUs), the deployment can scale to over 10K connections.

Distributed Transactions — Latency vs Scalability

Yugabyte DB is a horizontally write scalable database. This means that all nodes of the cluster are simultaneously active (as opposed to just one master, as is the case with Aurora). To achieve horizontally write scalability, the data is seamlessly split up into small chunks, called tablets, which are then distributed across all the nodes of the cluster.

When Yugabyte DB needs to perform a distributed transaction, it needs to perform writes across the different tablets, which end up being RPC calls to remote nodes. The upshot of this is that the database might have to perform RPC calls over the network in order to handle an end user’s transaction, which can affect both the latency and throughput as seen by the end user. With Amazon Aurora, the entire transaction is handled on the master node with no remote RPC calls.

This becomes an architectural tradeoff that is fundamental to the two designs, and thus needs some careful thought before picking one versus the other. But what do the raw performance numbers look like? In order to determine that, we performed a benchmark to insert 5M unique key-values into a database table with a secondary index enabled on the value column. There were no reads against the database during this period.

Analysis of Tradeoffs Using a Benchmark

Below are the results of this secondary index benchmark across these distributed PostgreSQL databases. These benchmarks write 5 million transactions (each of which write two keys as a transaction) with 128 writer threads. These benchmarks were performed with the standard setup outlined above.

Yugabyte DB needs 3–4 remote RPC calls before it can perform a distributed transaction involving multiple shards of the main table and the index (which is also modeled as a table). This results in a correspondingly higher latency and lower throughput. The write latency of a transaction in the above benchmark in Yugabyte DB is 22ms, while that of Aurora PostgreSQL is only 6ms. Additionally, the write throughput of a 3 node (16 vCPU) YSQL cluster is only 5.3K, while that of Aurora PostgreSQL is 20K.

Let us look at what happens when the time comes to scale the writes of the above workload. We have already discussed in a previous section that Aurora PostgreSQL can only scale to a maximum of 96 cores, or the write ceiling of the Aurora PostgreSQL database is capped at 120K transactions/second across all the transactions performed by the app and indexes on the various tables in the database. With Yugabyte DB, a 63 node cluster would deliver 120K transactions/sec, a 106 node cluster would deliver over 200K transactions/sec.

This means that Aurora PostgreSQL is a great choice if your database instance would never need to handle more than 120K transactions/sec. If future proofing for increasing scale is important, then Yugabyte DB is a better choice.

Note that the analysis in this section only applies for write transactions, reads are not affected by this analysis.

Future Work

We have a number of items we are working on.

Improve the performance of YSQL to be on par with YCQL, which is very achievable.

Change the connection handling architecture of YSQL. It currently spawns one process per connection, which can be a performance bottleneck. YCQL on the other hand, spawns one thread per connection and therefore can handle connection spikes much better.

We intend to make cluster-aware JDBC drivers the default for Yugabyte DB.

Run a TPCC benchmark against YSQL.

If you are interested in any of the above or other similar kinds of work, please reach out to us — we’re hiring!

What’s Next?