KhronosGroup/WebGL — pull request #202 by toji (2013-02-25 21:35:05+00:00)
Title: Allow the test runner to output command line arguments for reproducing failing tests

On suggestion from Remi Arnaud, A new line has been added to the top of the conformance test runners output files that describes the command line args needed to reproduce the failed tests from that run. For example, assuming several failures the top of an output file will now look like this:

'''
Executing command line args: --browser=Chrome --fast

To reproduce failures, run with the following args: --browser=Chrome --include=conformance/extensions/ext-draw-buffers.html,conformance/glsl/literals/float_literal.vert.html,conformance/glsl/misc/shaders-with-varyings.html,conformance/glsl/misc/shader-with-global-variable-precision-mismatch.html,conformance/more/functions/bufferDataBadArgs.html

-------------------

WebGL Conformance Test Results
Version 1.0.3 (beta)
...
'''

The "Executing command line args" shows the arguments passed to the test runner when this output file was produced.

In order to accomodate this functionality a new flag has been added to the conformance suite: failOnSkip. This flag defaults to "false", but if set to "true" any tests skipped by the "include" or "skip" flags will not be marked as failures but instead explicitly listed as "skipped". This will be the default mode for the test runner.

The 1.0.3 (Top of tree) and 1.0.2 test suites have been updated to support the new skipped test handling, but it has not been backported to 1.0.1 or 1.0.0 yet. They will continue to report skipped tests as failures.