bos/criterion — pull request #9 by basvandijk (2012-01-29 22:41:10+00:00)
Title: Support for comparing benchmarks against reference benchmarks

Hi Bryan,

The goal of my `vector-bytestring` [benchmark suite](https://github.com/basvandijk/vector-bytestring/blob/master/bench/bench.hs) is to compare `bytestring` with `vector-bytestring`. It contains benchmarks for all functions from `bytestring` and all similar functions from `vector-bytestring`. What I want to know is how much faster `vector-bytestring` is compared to `bytestring`. So currently I run the benchmarks, manually load them into a spreadsheet to calculate how much faster (in %) the `vector-bytestring` functions are and finally do some post-processing in emacs to prettify the result.

I would like to automate this and came up with the following solution:

There's a new combinator:

```haskell
bcompare :: [Benchmark] -> Benchmark
```

The first [bench](http://hackage.haskell.org/packages/archive/criterion/latest/doc/html/Criterion.html#v:bench) in the list (no matter how deep it's nested) is the reference benchmark. All other benchmarks are compared against that reference. The user can specify to write the comparisons to a CSV file using the ```-r``` (or ```--compare```) command line flag. The CSV file uses the following format:

```Reference,Name,% faster than the reference```

where the % is currently printed without precision (`%.0f`).

Note that a bcompare can contain any ```Benchmark```. So they can also contain bgroups and even other bcompares. The semantics of the latter are that:

* all the benchmarks within the nested bcompare are compared against the reference of the parent
* recursive step: all the benchmarks within the nested bcompare (except the first because it's the reference) are compared against the reference of the nested bcompare.

To use ```bcompare``` in ```vector-bytestring``` I only needed to make [a tiny change](https://github.com/basvandijk/vector-bytestring/commit/89bd9bd3616f6dc9a6a61dbbb091af154ab84199)!