Cassandra Data Migration with Dual Write Proxy - German Eichberger
Channel: The ASF
Date: 2021-10-11

Thank you to all of of our ApacheCon@Home 2021 sponsors, including:

STRATEGIC
---------------
Google

PLATINUM
--------------
Apple
Huawei
Instaclustr
Tencent Cloud

GOLD
-----------
Aiven OY
AWS
Baidu
Cerner
Didi Chuxing
Dremio
Fiter
Gradle
Red Hat
Replicated Cassandra Data Migration with Dual Write Proxy - German Eichberger

A presentation from ApacheCon@Home 2021
https://apachecon.com/acah2021

There are many databases implementing the Cassandra API and migrating from one to the other with zero downtime requires a dual write approach. The common approach is to modify the client applications to write to two API endpoints instead of one and potentially also read from the two endpoints and compare the results while write- and ttl-timestamp preserving offline migration tools (e.g. sstableloader) are used to move the other data. Modifying the client applications come at a huge cost and is not always feasible so many users are “stuck” with a particular Cassandra version or vendor.

This talk will introduce an Open Source dual write proxy under Apache License which will act as one endpoint for client applications but forward writes to two Cassandra API endpoints – thus making changes to client applications obsolete while leveraging the dual write approach. As an added benefit it will also monitor reads and provide reports showing discrepancies to aid in deciding when the migration is complete. Furthermore, we will review multiple real life data migrations using this proxy and the accompanying offline migration method and share lessons learned.

This talk is applicable to Cassandra users who want to upgrade from an older Cassandra version to 3.X or 4.X but also to users who want to switch to or from any of the cloud based Cassandra API offerings.