FoundationDB/sql-layer — pull request #1261 by nathanlws (2014-04-17 18:55:23+00:00)
Title: Adjust MT synchronization for deterministic marks

MTs have been failing intermittently due to out of order expected time mark results. Adjust synchronization points to make them actually deterministic.

The general flow is a couple of threads attempting to exercise a certain sequence. This is done by setting up a couple barriers and then recording "time marks" when a thread hits a certain stage. Take a careful look at each of the helper functions and ensure that the syncs allow for a completely deterministic mark order.

For a little extra convincing, inject random sleeps before or after marking at various points. For each test run the test builder now selects a time and set of stages where a 50ms sleep will be inserted. This was quite good in sussing out orders that weren't always correct. To help with reproducibility, a jUnit hook was added to print out random parts upon failure.

This branch has ran successfully a few hundred times locally and fifty times via the sql-layer-mt job (on cycloeps) on Jenkins.