bos/criterion — Issue #3: Possible error in data analysis
Author: Shimuuar
Created: 2011-10-22 18:05:35+00:00
State: open

I think there is an error in criterion's data analysis. Here is simplified description of algorithm as I understand it. For simplicity I will ignore discreteness of clock.

1. Clock call cost is estimated.
2. For every benchmark N samples are collected
3. These samples are corrected for timing overhead and somehow averaged. Exact nature of averaging is not significant for reasons stated below.

The problem is that clock call cost is measured quantity and this measurement have error (σ(t) from now on) and it's never taken into account. This error corresponds to shifts of timing distribution as whole and couldn't be eliminated. No averaging procedure could detect such shifts. So error for benchmark couldn't be less than σ(t). It shouldn't be significant for long-running functions. But it is significant for function which take same or less time to complete as getPOSIXTime. mwc-random's benchmarks should be affected.