Project: ScalarDB (App Definition and Development / Database)
File: TransactionManagementOnCassandra.pdf

Transaction Management on Cassandra10 Sep,2019 at ApacheCon NA 2019Hiroyuki YamadaCTO/CEO at Scalar, Inc.1
© 2019Scalar, inc.
Who am I ?•Hiroyuki Yamada–Passionate about Database Systems and Distributed Systems–Ph.D. in Computer Science, the University of Tokyo–IIS the University of Tokyo, Yahoo! Japan, IBM Japan
2© 2019Scalar, inc.
Agenda•What is Scalar DB•Transaction Management on Cassandra•Benchmark and Verification Results
3© 2019Scalar, inc.
Maybe You Don’t Need ACID Transactions•ACID transactions are heavy–Especially when data is distributed•One of other solutions:–Make operations idempotent and retry them if atomicity is not required
4© 2019Scalar, inc.
What is Scalar DB•A library that makes non-ACID distributed databases ACID-compliant–Cassandra is the first supported distributed database
5
https://github.com/scalar-labs/scalardb
Transaction management, Recovery management, Java API© 2019Scalar, inc.
System Architecture with Scalar DB and Cassandra
6Cassandra nodesAchieves one-copy Serializable Transactions
Web ApplicationsClientprogramsScalar DB
DataStaxJava DriverEnd UsersHTTPCommandexecution
Key key = new Key(new TextValue(“id”, ”1”));Result result = db.get(new Get(key));// do something with resultPut put = new Put(key).(…);db.put(put)Scalar DB© 2019Scalar, inc.
Key Characteristics•Non-invasive approach–Any modifications to the underlying database are not required•Highavailability–Available as long as quorum of replicas are up–C* high availability is fully sustained by the client-coordinated approach •Horizontal scalability–Throughput scales linearly–C* high scalability is fully sustained by the client-coordinated approach •Strong Consistency–Replicas updated by transactions are always consistent and up-to-date7© 2019Scalar, inc.
Transaction Management on Cassandra -Introduction•Based on Cherry Garcia protocol [ICDE’15] –Requires minimum set of features such as linearizable conditional update and the ability to store metadata•Scalar DB is one of the applications of the protocol–Use LWT for Linearizability–Manage transaction metadata in user record space•Implement enhancements–Protocol correction–No use of TrueTimeAPI–Serializable support (SI is the default isolation level)8© 2019Scalar, inc.
Transaction Metadata Management•WAL (Write-Ahead Logging) records are distributed  
9Application dataTransaction metadataAfter imageBefore imageApplication data(Before)Transaction metadata(Before)StatusVersionTxIDStatus(before)Version(before)TxID(before)
TxIDStatusOther metadataStatus Recordin coordinatortableUser/ApplicationRecordin user tables
Application data(managed by users)Transaction metadata(managed by Scalar DB)© 2019Scalar, inc.
Transaction Protocol -Overview•Optimistic concurrency control•Similar to 2 phase commit protocol–Prepare phase: prepare records–Commit phase 1: commit status–This is where a transaction is regarded as committed or aborted in normal cases–(Commit phase 2: commit records)•Lazy recovery–Uncommitted records will be rollforwardedor rollbacked based on the status of a transaction when the records are read10© 2019Scalar, inc.
Transaction Protocol By Examples –Prepare Phase
11Client1Client1’s memory spaceCassandraReadAtomic conditional update (LWT)Update only ifthe version is the version I readFail due tothe condition mismatchUserIDBalanceStatusVersion1100C5TxIDXXX2100C4YYY180P6Tx12120P5Tx1Tx1: Transfer 20 from 1 to 2
Client2UserIDBalanceStatusVersion1100C5Client2’s memory spaceTx2: Transfer 10  from 1 to 2TxIDXXX2100C4YYY190P6Tx22110P5Tx2UserIDBalanceStatusVersion1100C5TxIDXXX2100C4YYY© 2019Scalar, inc.
Transaction Protocol By Examples –Commit Phase 1
12Cassandra
UserIDBalanceStatusVersion180P6TxIDTx12120P5Tx1StatusCTxIDXXXCYYYAZZZCTx1Atomic conditional update (LWT)Update ifthe TxIDdoes not existClient1 with Tx1© 2019Scalar, inc.
Transaction Protocol By Examples –Commit Phase 2
13Cassandra
UserIDBalanceStatusVersion180C6TxIDTx12120C5Tx1StatusCTxIDXXXCYYYAZZZCTx1Atomic conditional update (LWT)Update status ifthe record is prepared by the TxIDClient1 with Tx1© 2019Scalar, inc.
Failure Handling by Examples•If TX1 fails before prepare phase–Just clear the memory space for TX1•If TX1 fails after prepare phase and before commit phase 1 (no status is written in Status table)–Another transaction (TX3) reads the records and notices that the records are prepared and there is no status for it –TX3 tries to abort TX1  (TX3 tries to write ABORTED to Status with TX1’s TXID and rolls back the records)–TX1 might be on it’s way to commit status, but onlyone can win, not both•If TX1 fails (right) after commit phase 1–Another transaction (TX3) tries to commit the records (rollforward) on behalf of TX1 when TX3 reads the same records as TX1–TX1mightbeonit’s way to commit records, but only one can win, not both14© 2019Scalar, inc.
Benchmark Results
15
Workload2 (Evidence)
Workload1 (Payment)
Each node: i3.4xlarge (16 vCPUs, 122 GB RAM, 1900 GB NVMeSSD * 2), RF: 3•Achieved 90 % scalability in 100-node cluster (Compared to the Ideal TPS based on the performance of 3-node cluster)© 2019Scalar, inc.
Verification Results•Scalar DB has been heavily tested with Jepsen and our destructive tools–Note that Jepsen tests are created and conducted by Scalar•It has passed both tests for a long time•See https://github.com/scalar-labs/scalar-jepsenfor more detail
16Jepsen Passed© 2019Scalar, inc.
Other Contributions for Apache Cassandra from Scalar•GroupCommitlogService–Yuji Ito–Group multiple commitlogwrites at once–CASSANDRA-13530•Jepsen tests for Cassandra –Yuji Ito, Craig Pastro–Maintain with the latest Jepsen–Rewrite with Alia clojuredriver–https://github.com/scalar-labs/scalar-jepsen•Cassy–A simple and integrated backup tool–Just released under Apache 2–https://github.com/scalar-labs/cassy17© 2019Scalar, inc.
Cassy: A simple and integrated backup tool
18
•Required to take a transactionallyconsistent backup© 2019Scalar, inc.
Future Work•DataStax driver 4.x support•Cassandra 4.x support–Hopefully nothing needs to be done•Other C* compatible databases integration–Scylla DB (waiting for LWT)–Cosmos DB (waiting for LWT)•(HBase adapter)
19© 2019Scalar, inc.
Questions ?
20© 2019Scalar, inc.
Optimization•Prepare in deterministic order–=> First prepare always wins
21TX1: updating K1 and K2 and K3TX2: updating  K2 and K3 and K4H: Consistent hashingK1K2K3K2K3K4Always prepare in this order !!!(otherwise, all Txsmight abort. Ex. TX1 prepares K1,K2 and TX2 prepares K4,K3 in this order)H(K2)H(K1)H(K3)H(K4)© 2019Scalar, inc.
Snapshot Isolation•Strong isolation level but weaker than Serializable–Similar to “MVCC”–Oracle’s most strict isolation level (it’s called “Serializable”)•Read only sees a snapshot (=> non blocking reads)•Mostly strong enough but there are still some anomalies
22© 2019Scalar, inc.
Anomalies in Snapshot Isolation•Write Skew, Read-Only Transaction•Write skew example:–Account balances: X and Y (assume family account)–Initial state: X=70, Y=80–Constraint: X + Y > 0–TX1: X = X –100, TX2: Y = Y -100–H: R1(X0, 70) R2(X0, 70) R1(Y0, 80) R2(Y0, 80)W1(X1, −30)C1 W2(Y2, −20)C2
237080X0Y070-100 -> -3080XYTX17080-100 -> -20XYTX2Update succeeds without conflictin Snapshot IsolationOk for the constraintOk for the constraintUpdate XUpdate Y-30-20
© 2019Scalar, inc.
Serializable Support•Convert all reads into writes (writing the same value) in a transaction
24© 2019Scalar, inc.
Protocol Correction•Commit records by non-atomically–=> NO!!!•Someone else have already did it and have started and prepared a new transaction
25TX1TX2Start and Prepare ACommit StatusCommit A (without knowing A with TX1 is already committed, and A is overwritten by a new TX2)
Read A and commit A on behalf of TX1Start and Prepare ACommit StatusCommit A