drothlis/stb-tester — pull request #76 by wmanley (2013-12-09 17:49:55+00:00)
Title: report: Replace percentage with percentage confidence interval in summary

As you run more repetitions of a test you get a better idea of how reliable the test is. A percentage is useful to describe a failure rate independent of the number of runs but doesn't include information on how reliable that percentage is likely to be.

This commit replaces the percentages in the report with percentage ranges called "confidence intervals". This means that instead of 43/430 being a failure rate of 10% it will read 7.5% - 13%. This means that there is a 95% chance that the "true" reliability of the test falls between 7.5% and 13% rather than the potentially misleading 10%.

![confidence-interval](https://f.cloud.github.com/assets/494659/1707207/417cb344-60fa-11e3-9edc-cfece317ae68.png)

We use the [Wilson Score Interval][1] to calculate the confidence interval. This assumes a binomial distribution (e.g. test pass/fail results are independent of each other).

This commit also introduces a doctest like test runner for javascript which I wrote to test my wilson_score_interval_95 function. It depends on [seed][2] to run the javascript headless but could probably be fairly easily extended to run with nodejs or similar instead.

Possible future work:

* Include a further breakdown in the report summary by test exit status
reason.
* Include a Chi-Squared Test for comparing the results with different
filters applied to answer questions like "has reliability improved
between these two versions of software?".
* Write tutorial documentation to help users understand the statistics
and how to interpret test results.

See also:

* [Website for calculating confidence intervals][4]
* [Evan Miller's excellent "Statistical Formulas For Programmers"][5]

[1]: http://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval#Wilson_score_interval
[2]: https://wiki.gnome.org/Seed
[3]: http://www.evanmiller.org/ab-testing/chi-squared.html
[4]: http://vassarstats.net/prop1.html
[5]: http://www.evanmiller.org/statistical-formulas-for-programmers.html