OpenImageIO/oiio — pull request #241 by lgritz (2012-03-06 07:03:19+00:00)
Title: Review: testsuite overhaul

Weekend project...

This overhaul of the runtest.py and individual run.py files finally fixes a longstanding annoyance -- that the tests are run directly in testsuite/... and also all sorts of waste is left behind.

With this change, the testsuite is first copied to build/ARCH/testsuite, and run from there, so regular oiio/testsuite is considered read-only by the tests themselves. This means that a "make clean" clears all the cruft from the tests, nothing left behind. In the process of this change, much of the repetitive code in the individual testsuite/*/test.py files has been moved to runtest.py, thus making the run.py files in general much shorter and easier to understand.

I also fixed up a few tests and removed some of the os-to-os variance, and in the process of making those things work well, I made a number of other small changes that are part of this review (but for which I will keep separate individual commits, in case we want to cherry pick them):

* oiiotool -hash now prints the hash of each input image (like iinfo -hash always did)
* oiiotool is better about printing error messages and exiting when it comes across an input image that it can't read.
* idiff (and oiiotool --diff) no longer print NaN nonsense for 0x0 images.
* testtex now supports -d (like iconvert and oiiotool) to set the desired data format for the output.
* idiff and oiiotool --diff, when NOT given the -v (verbose) flag, will only print "PASS" for non-failing comparisons. Only failing images (and all images if -v is used) will print the pixel statistics. This is helpful if you want to raise the thresholds to allow a little bit of LSB disagreement from platform to platform -- the output logs will still not match if each one prints its (passing) differences. It's better for a pass-with-threshold to just print "PASS" without the stats, so that all platforms will have matching logs.

I've tested this on OSX and Linux. It would be great if somebody could test on Windows and let me know if I've broken anything horribly (though, truth be told, I have no idea if the testsuite ran reliably on Windows even before this overhaul).

If nobody objects, I'd like to backport this to 1.0 release as well, since it's so much nicer to have the tests execute in build (and this fixes several broken tests).

For you OSL-ers out there, after this is all approved, I will do the analogous simplification and overhaul to the testsuite in OSL, it could stand to have exactly the same improvements.