Taking advantage of Aerospike’s XDR Enhancements in V5.x to reduce messaging costs
By: Reuven Kaswin
Tags: Database, Aerospike

With Aerospike version 5.x, the Cross Data Center (XDR) feature was rearchitected from scratch to make it more customizable and resilient, making it much more useful. Additional functionality is forthcoming later this year to enhance the customizability even further.

In prior Aerospike versions, connectivity from a source database to multiple destination databases were interdependent. If there was an issue (whether network, latency, or throughput issues), then shipment to all destinations were affected. With V5.x, shipment to multiple destinations has been completely decoupled.

A couple of excellent blog posts covering XDR as well as Multi-Site Cluster functionality were written by Srini Srinivasan, the Chief Product Officer of Aerospike and by Sunil Sayyaparaju, a VP of Engineering at Aerospike that get into the nuts and bolts of XDR in Aerospike 5.x. While those blogs cover the nuts and bolts of the redesigned XDR, this post is intended to discuss a use case for which XDR would be ideal and how it may be applied.

“Real-time” Data Hub

Consider a hypothetical web retailer with 100 million SKUs (unique item records) in their Aerospike Data Hub DB utilizes that database for millions of real-time pricing requests, provided to customers via their web site. Pricing is updated on an hourly basis for 10–20 million of those records, but at peak periods, it is possible that all records would be updated hourly. Those records are coming from multiple source applications systems. Obviously, those updates need to be completed within that hour, prior to the commencement of next hourly batch.

Each of those SKU updates need to be transmitted to multiple applications for AI/ML, local (regional) access, and other use cases. This is where XDR comes into play. The applications that are able to handle the same throughput as the Data Hub will receive data at the same rate that the Data Hub is receiving them, while slower applications will receive the data at a slower rate.

Why use XDR to ship data to multiple applications instead of using a streaming platform such as Apache Kafka?

One important reason is that XDR will only ship the current data to another system. If, for example an application is down for hours or days, once the application is once again online, only the latest (current) attributes of records will be transmitted via XDR. The alternative of using a streaming platform would require the application to update records that have changed multiple times since the application was down multiple times also. Obviously, this would also involved wasted data transfers, which may be expensive between cloud-based applications or limited as in lower bandwidth WAN messaging. For example, the current Amazon EC2 pricing for messaging between Availability Zones or Regions range between $.01 and $.02 per GB. As some of our customers have painfully discovered, those pennies can quickly add up to large bills.

Another major reason it to provide filtering of data at the Aerospike source database such that each downstream application will only receive records/attributes that are of interest to that application. This is often a core requirement based upon national data transfer (such as GDPR) limitations. Again, considerations of the cost of messaging between different cloud-based applications and/or WAN messaging bandwidth must also be part of a TCO/latency discussion. These costs can add up very quickly.

Aerospike’s XDR implementation includes the capability of DCs to automatically forward messages between datacenters. This provides another cost-saving and bandwidth saving opportunity as one Aerospike database in a datacenter can be used as a local hub, forwarding messages to other databases (with optional filtering) to other databases within that datacenter.

Aerospike XDR Data Hub Features

Some of the features that Aerospike provides that provide a basis for their “Data Hub” functionality are as follows:

Dynamic configuration of destination databases

Filtering of data by set (table) or bin (attribute) on a per-destination basis

Shipment of latest set/bin values to destinations under all circumstances

Compression/pipelining of messages between data centers

The Bottom Line

As more and more applications move to the cloud, the messaging costs have been growing significantly, sometimes being even greater than the servers costs. Taking advantage of the Aerospike XDR features with an appropriate architecture can minimize those costs.