avieth/hedis — pull request #2 by avieth (2014-10-12 16:55:08+00:00)
Title: Benchmarking: use criterion, remove concurrency

I'm trying to determine whether a pull request to infomatrikr/hedis https://github.com/informatikr/hedis/pull/19 has any implications on performance. The existing benchmarks were not up for the task. If I understand them correctly, it worked like so:

- There are 50 connections which are set up before any clocks are checked.
- To time an action, we divide the work among the 50 clients, but only one client does any work at a given time, because the start MVar could only be taken by one client thread.
- The requests per second statistic included the noise of pure computation not relevant to hedis like MVar operations and replicateM calls.

This suite gave us some idea of throughput (even though the clients were not acting concurrently as they should have), but we're also interested in the latency of smaller interactions like firing up a connection, running a few gets, and returning. This new suite accomodates that.

It's still not clear to me how best to benchmark a redis client. Suggestions needed.