josephg/Chipmunk-js — Issue #3: bench.js: easy/fast way to answer the question 'did I regress?'
Author: nornagon
Created: 2012-02-25 09:40:51+00:00
State: open

Currently the benchmarks take upwards of 15 minutes to run. It's really hard to determine, for a given change, whether a performance regression was introduced.

Possible solutions:

1. only run each benchmark 3 times, instead of 9. This might lead to less stable numbers, but if it doesn't then it's a good solution.
2. define a 'fast' subset of benchmarks that's reasonably representative. You've been tracking the benchmarks over time, so it shouldn't be too hard to pick the few that have reliably indicated overall performance (least-squares analysis perhaps? :))
3. introduce a single large 'complete' benchmark. Even if it took 20s to run each time, that would only be 9*20s = 3 minutes total.

It would also be nice to be able to easily compute baseline numbers, e.g. by 'node bench.js --did-i-regress' performing a git clone into a temporary directory and running the benchmarks at origin/master, then running with your changes.