Scaling high throughput Writes and Reads
By: Sreesha S Kuruvadi
Tags: Design Thinking, Software Development, Scale, System Design Interview

GatewayR- Entry point for Users who consume via Reads.

GatewayW- Entry point for Transactional Users.

Twemproxy- A daemon present on each of the API VMs which helps map the incoming data to it’s repsective redis shards.

LB- Load balancers help distribute the load across multiple VMs.

VMi — APIs are horizontally scalled by adding multiple VMs under a load balancer.

Subscribers-Vertically and horizonatally scaled stateless VMs which listen to push events from IPC mecahnisms. They play a crucial role in keeping data consistent across the two parts of the above system, or all parts of an n-partitioned system.

Understanding the nature of the User — Users can be multi-natured

Transactional Users ( High throughput writes )

These users run DML and DDL qieries on persistent stores like PostgresSQL or MySQL

High through put writes will require updates to be propagated throughout the system. A single update can trigger update in multiple parts of the system.

These updates need to the reach Users who consume via reads in near-realtime.

Users who consume via Reads ( High throughput reads )

Requires low latency highly available APIs

Near-realtime reads

In the above diagram these Users acces the system resources through different regions architected different from each other and is represented by the Blue ( right top) and Pink ( bottom left) regions respectively.

Notable points

While nature of the users differ, the systems differ in a similar manner. This warrants a need for a single source of truth for generated data, which is also consumed

for generated data, which is also consumed PostegreSQL and MySQL are used as persistent stores, albeit one is configured for large number of writes and the other for reads.

With a single source of truth comes the question of scaling, where availability and partition tolerence decreases. The following explains the reasons why MySQL and PostgreSQL are being used separately and form the persistent stores separately for Reads and Writes respectively.

Consistency vs Availability vs Partition Tolerence

Consistency

Ensuring consistency means that all nodes in the system see the same data at the same time.

Availability

Availability means that the systems are responsive with a high uptime

Partition Tolerence

While these systems use IPC for communication, the system continues to operate despite arbitrary message loss or failure of part of the system.

The above system is inherently partition tolerent since the reads and writes are happening from two different systems and being scaled individually.

High Consistency — In the above system, low latency in-memory redis pub/sub channels, updated the REDIS cluster and MySQL simultaneously.

Redis clusters have masters and slaves which bring in eventual consistency where updates are propagated through slaves. The update propagation time among slaves is directly proportional to the number of slaves, the update throughput and the amount of data being handled.

Sharding with master redis and slaves would provide the perfect combination for availability and consistency

Sharding ensures data is spread across nimble clusters which have a lesser individual throughput which warrants lesser number of slaves that need to come into sync. Hence ensuring faster eventual consistency

High Availability

Replication among slaves ensures high availability for reads for each of the sharded redis masters.

High Partition tolerence

Cache cluster

Represented by the brown circle at bottom right

Cache cluster contains sharded masters with slaves — Slaves can re-elect a new master in the event of a failure without loss of data.

Sharding helps the system with high resilience since the systen is under lesser stress due to reads and rights spread across multiple shards

In-case due to a network failure, if an update to a shard fails, one or more shards will end-up with stale data. Handling updates to stale data without redundant updates by multiple VMs under GatewayR and GatewayW can be done using mutext locks.

Data flow between the PostgreSQL and MySQL

IPC methods may contain retrys to help updates propagate through the system immediately when the system recovers after a catastrophic failure. RabbitMQ used above is a highly available queue which buffers updates from write APIs and provides persistence for retry in case of network failures or subscribers going down.

Keeping MySQL in sync with PostgreSQL is of utmost neccessity, with PostgreSQL being the source of truth, any network failures/node failures can cause descrepency in the data consumed at GatewayR with that generated at GatewayW

Realtime Updates

With the system seperated into two parts and scaled independently, realtime updates requires their own set of systems and scaling procedures. In the above example, highly available in-memeory redis pub-sub channels are used to propagate updates through to the cache cluster in near real time, while the DDL and DML APIs at GatwayW update the data to PostgreSQL and MySQL eventually.

are used to propagate updates through to the cache cluster in near real time, while the DDL and DML APIs at GatwayW update the data to PostgreSQL and MySQL eventually. Further sharding these channels , for example per region can help scale the system better and reduce latency even further.

, for example can help scale the system better and reduce latency even further. An incoming write to PostgreSQL at GatewayW, will trigger a push through redis realtime in-memory pubsub channel which is consumed by the subscriber which inturn updates the cache cluster in near-realtime. This update is immediately avilable to a user via GatewayR on the refresh of a web-page or through an event via socket-io.

This update is immediately avilable to a user via GatewayR on the refresh of a web-page or through an event via socket-io. A write to PostgresSQL and a data push to Rabbit MQ, followed by a write to MySQL happens eventually through the system which ensures PostgreSQL and MySQL are in sync.

Do note that redis in-memory pub-sub events propagate at low latencies as compared to that of RabbitMQ.

Refreshing Data

CRONs which run periodically, keep the two persistent stores in sync and the cache updated with the most recent data, are a key part of the above system in case, sequential network failures prevent updates from propagating from one end to the other.