FoundationDB/sql-layer — pull request #1083 by nathanlws (2013-12-05 04:29:48+00:00)
Title: Concurrent DML during online DDL

All writes (insert, update and delete) are now properly checked and/or propagated to the new locations. About 65% of the diff is new MTs and test support code, but that doesn't make the diff any smaller. The individual commits are pretty targeted.

High level flow:
- Bump table versions at each start and finish of online DDL
- Concurrent Sessions get a map containing all table IDs they have written to
- Prior to commit, check transactional version of those table IDs with the global
- Abort commit if changed (NB: See comments in 2fc757d for more details)
- OnlineHelper is now a RowListener, transforms and applies concurrent DML with respect to new in-progress DDL
- Concurrently handled DML is tracked, by (tableID,hkey) as to not be double applied by DDL session

A number of the changes were to facilitate locking as need by Persistit to prevent write skew. This will nop on FDB, of course.

The concurrent DML HKey tracking isn't ideal. A system table would be more straight forward but couldn't support maximum sized HKey (long + int + ordinal overhead). Currently, they are placed directly into a location by the SchemaManager to avoid that. An alternative would be to drop the max hkey size at DDL time by 20 bytes or so to facilitate this. These entries are only present while DDL is in progress so it isn't an upgrade issue if we want to change it in the future.

One last wart is the OnlineDDLMonitor in BasicDDLFunctions. The `sync`-ing the test builder/MTs do makes it not so easy to have it be a direct Service injection so I used a static field. I could add another level of indirection if that's too objectionable.

As hinted at, there are extensive new MTs that exercise all of the new code's obvious interaction points. Both positive and negative cases, including delicacies like orphaning and adoption. They fail rapidly if the DDL session doesn't do locking or table versions aren't checked, for example, so I'm rather happy with them.

The problem, as always, would be areas that *aren't* covered. As we discussed, the next branch will short-circuit `OnlineHelper#concurrentDML()` to disallow writes by default until our confidence is higher.