mozilla/browserid — pull request #2272 by jrgm (2012-08-13 09:01:13+00:00)
Title: optionally use node-cover when running npm test for backend code coverage data

An experiment with running backend tests with a code coverage tool. DO NOT MERGE this yet. I want to convince myself that the numbers make sense, although so far they (mostly) do. But I thought I'd put this up here for people to ponder and try out (and for me to refer to when I file some issues where we are missing tests).

- No additional setup is required beyond `npm install` and setting WITH_COVER=1

- this will only have effect if WITH_COVER is set in the environment. At the end of a run, a ./cover_html directory is generated with nice annotated html reports per top-level or ./lib file used. Example: `WITH_COVER=1 WHAT_TESTS=back npm test`

- Caveat on the html reports: the %lines number overstates coverage since it includes comments and blank lines. The %blocks is more useful in figuring out where more tests would be useful

- uses `node-cover` which bundles `esprima` for js munging.

- this PR alters `run_locally.js` to start the daemons wrapped in the `cover` command

- also adding `rimraf` to packages.json to remove the generated .code_coverage directory between runs

- to get coverage data done correctly for `bin/router` I had to register with `require('../lib/shutdown').handleTerminationSignals`

- however when I tried do the same thing for handleTerminationSignals for `bin/static`, the tests started failing. I should look more into shutdown to figure this out. For now, `bin/static` usage of libs will be underreported.

- running with WITH_COVER=1 costs 20% on wall-clock times, which is relatively cheap

- currently back,back_mysql,front all pass for me either WITH_COVER or not. (Although running with WHAT_TESTS=front isn't particularly interesting probably. Also, some wsapi that are not called at all by `back` are called for only for failure cases with `back_mysql` (`./tests/stalled-mysql-tests.js`))

- I've only run this on osx, so I should try on linux (and win32 where I have a limping install going now).