ariatemplates/ariatemplates — pull request #3 by jakub-g (2012-06-19 11:50:43+00:00)
Title: Test runner improvements

This pull request is to suggest several small improvements to our test runner, so manual testing at the end of each sprint is less painful.

My proposals:
1) When a test suite is finished, it should be very clearly indicated, e.g. by changing the background color of the page. That way we can easily see whether the suite is finished, or it hanged on some of the tests. Looking for the status text requires more attention.

2) We should extract the test cases that
- are somehow 'problematic'
- take a long time,
- require focus,
and move them to be launched at the end of the suite (80/20 rule). It's easier to manually skip the tests at the end of the suite, than to check every 20 seconds if the suite is running.

3) Perhaps we should add a query string option, that instead of running the whole suite, would list all the tests (not recursive, only the top-level with respect to the classpath) as a clickable URLs?

Something like http://aria/.../test/?autostart=false

As we know, it's not advisable due to some reasons to launch the whole test suite from the root (especially, when it breaks in the middle, you don't know where to restart). Current way of doing it is, from what I see, to open a test suite JS file, and copy the classpaths of the tests from sub-suites (tedious).

For now, I've listed the classpaths as links in our internal Aria_Templates_Delivery_Guide, but this has a drawback that it must be kept in sync with the JS file, and we all know at some stage someone will forget to update both at the same time. It's safer to add a parameter to the test runner.

4) (partially overlapping 3) Basically, I think it would be good to have a pre-index page in the test runner (parametrized by ?classpath=...), on which you will have the links to
-- start the mentioned suite
-- to list the tests in the mentioned suite
-- additionaly, somehow choose whether the previous should be done with "runIsolated=true" or not.

Go on with your thoughts on this, and suggest your improvements! Feedback very welcome.