pmembrey/cakedb — pull request #21 by AlexandreBeaulne (2013-01-11 02:18:30+00:00)
Title: Make the file write delay customizable by the user

This pull request makes two changes to one of the core file of CakeDB, src/cake_stream.erl.

Before I describe the details of the two modifications, let me justify the reason for the changes. Dissatisfied with the rigorousness and efficiency of the PropEr tests I made earlier, I've been thinking long and hard about how to make them more relevant and faster.

The biggest hurdle I faced was that in the current state of affairs, there is a long (~8-12 secs) and unpredictable delay between the time a driver send data to Cake and the time it is written in the binary data files. There is a good reason for this, grouping some data together before writing to file reduces the number of file operations and consequently make CakeDB more reliable by potentially reducing data corruption.

However, that delay made systematic, randomly-generated testing (e.g. PropEr) extremely hard if not impossible. The existing unit tests (for the Java and Erlang drivers) had sleep instructions (up to 12 secs) placed strategically to give the time to Cake to flush before performing the test queries. However PropEr generates hundreds, if not thousands, of test instructions not only with random inputs but also in random sequences. If sleep instructions are put in every query, the tests takes unacceptable amount of time to run, and that's what occurring right now with src/cake_protocol_statem.erl.

My solution consists of two relatively small fixes:

(i) Creating an environment variable, write_delay, in src/cake.app.src, that default to the current value, ten seconds. Nothing will change for servers running CakeDB, however developer wanting to run tests can overwrite the value with a lower delay.

(ii) Right now, if the Stream does not have the clear signal from the Writer, it waits up to five seconds to try again. I moved a piece of code in src/cake_stream.erl so that the Stream sends received data as soon as it receives the clear signal from the Writer.

These changes might appear of little utility, but I believe they will allow me to significantly improve the state of PropEr testing. For a sneak peak of what I am talking about, one can look at the branch [proper_testing](https://github.com/AlexandreBeaulne/cakedb/tree/proper_testing) in my personal repo.