bos/criterion — pull request #10 by tibbe (2012-02-29 18:40:57+00:00)
Title: Run the garbage collector before each benchmark

This makes sure that garbage created by previous benchmarks doesn't affect the current benchmark.

Here's an example of running an allocation heavy benchmark, `insert/Int`, followed by an allocation free benchmark, `lookup/Int`, prior to my change:

~~~~
warming up
estimating clock resolution...
mean is 1.704824 us (320001 iterations)
found 1648 outliers among 319999 samples (0.5%)
1206 (0.4%) high severe
estimating cost of a clock call...
mean is 53.82255 ns (10 iterations)

benchmarking insert/Int
mean: 1.211003 ms, lb 1.124407 ms, ub 1.356176 ms, ci 0.950
std dev: 560.4286 us, lb 343.8345 us, ub 771.3425 us, ci 0.950
found 6 outliers among 100 samples (6.0%)
6 (6.0%) high severe
variance introduced by outliers: 98.928%
variance is severely inflated by outliers

benchmarking lookup/Int
mean: 194.1801 us, lb 192.9880 us, ub 195.5927 us, ci 0.950
std dev: 6.588454 us, lb 5.524500 us, ub 8.255747 us, ci 0.950
found 5 outliers among 100 samples (5.0%)
2 (2.0%) high mild
2 (2.0%) high severe
variance introduced by outliers: 29.696%
variance is moderately inflated by outliers
~~~~

And after my change:

~~~~
warming up
estimating clock resolution...
mean is 1.709502 us (320001 iterations)
found 1691 outliers among 319999 samples (0.5%)
1197 (0.4%) high severe
estimating cost of a clock call...
mean is 51.61498 ns (11 iterations)
found 1 outliers among 11 samples (9.1%)
1 (9.1%) high mild

benchmarking insert/Int
mean: 1.206301 ms, lb 1.120921 ms, ub 1.352673 ms, ci 0.950
std dev: 557.1362 us, lb 361.4757 us, ub 777.2095 us, ci 0.950
found 6 outliers among 100 samples (6.0%)
6 (6.0%) high severe
variance introduced by outliers: 98.928%
variance is severely inflated by outliers

benchmarking lookup/Int
mean: 174.1413 us, lb 173.5130 us, ub 174.8956 us, ci 0.950
std dev: 3.503936 us, lb 2.863068 us, ub 4.444968 us, ci 0.950
found 11 outliers among 100 samples (11.0%)
4 (4.0%) low mild
5 (5.0%) high mild
2 (2.0%) high severe
variance introduced by outliers: 13.244%
variance is moderately inflated by outliers
~~~~

Notice how `lookup/Int` is now faster, presumably because fewer garbage collections (in particular major ones) happened during the run.