jdhenke/pad — pull request #13 by jdhenke (2014-04-25 23:34:31+00:00)
Title: PhantomJS Testing

Here's where I'll be committing my incremental work towards an end-to-end testing infrastructure.

## Tentative Outline

The current plan is to use PhantomJS to populate many headless clients, which each instantiate their own pad javascript clients.

Some clients will write, some will only read. To start, I'd like to test for

**eventual consistency** - basically after pausing for a decent amount of time, are all clients the same?
**commit latency** - for each commit record when it is

* sent to the server
* received by the client
* applied to the state of each client

This should allow us to have a distribution of min, average and maximum propagation time for all commits, both for receipt and application of the commit.

## Concerns

**We should ask the course staff about these issues.**

* with all this being on one machine, the *machine* might be the bottleneck... should probably look into spinning all these things up on different machines. Testing latencies would be hard though without synchronized clocks...
* failures - we need a well defined "failure API" of sorts, so it can be automatically tested.
* testing better consistency guarantees - i think eventual consistency isn't very strong; we want reasonable semantics while things are being concurrently updated; how to define is hard though.