tc39/test262 — pull request #67 by smikes (2014-08-01 11:56:56+00:00)
Title: Supply error names to in place of flags: negative

There are 313 tests which use the notation `flags: [negative]` and do not specify an expected error name. (There may be more that use `flags: \n - negative` but I am starting with a naive grep).

Here is a PR that begins to convert those tests to use an explicit error type, starting with the `bestPractice` directory. The error type was determined by removing `flags: [negative]` and running against V8 and SpiderMonkey; if those two engines threw the same type of error, e.g. a `SyntaxError` then I amended the test to expect that error with `negative: SyntaxError`.

This method has the potential to spuriously move an engine from "conformance" to "non-conformance" if the spec has underspecified the type of error that should be thrown. If I change the test suite to expect `SyntaxError` on the basis of two implementations, and some other implementation validly chose `TypeError`, then that implementation will suddenly, retroactively, appear to be failing the test suite. I think the risk of this is low, but it's worth calling it out.

I'd like some opinions please, about --
- whether this work is worth doing
- whether the approach sounds reasonable
- what level of testing should be performed before allowing this to be patched into master (all modern browsers and standalone engines? older browsers?)

/cc @anba