wesm/vbench — Issue #1: Use unittests as benchmark-cases and benchmarks as unittests
Author: JanSchulz
Created: 2011-12-19 13:23:19+00:00
State: open

Some ideas:

There are already lots of functions to benchmark: unittests. It would be great if your code would run them and just add a datapoint for each testcase per version. It would have some false results due to changing unittests, but on the other hand you gain lots of datapoints quickly. You could even add a html output which outputs the results in a table of unittests per version (green/red for better/worst results).

Also the reverse: running benchmarks as part of the unittests and failing if the result degrades (too much). Unfortunately, I've no idea how to get earlier results to compare to (same hardware and CPU load).

Eclipse has such a tool:
http://wiki.eclipse.org/Performance
http://dev.eclipse.org/viewcvs/viewvc.cgi/org.eclipse.test.performance/doc/Performance%20Tests%20HowTo.html?view=co