imageworks/OpenShadingLanguage — pull request #80 by lgritz (2011-10-14 18:52:57+00:00)
Title: Review: make testsuite test both unoptimized & optimized

The testsuite has always run with a low runtime optimization setting, on the theory that if the code in the test shaders gets optimized away, it's not really testing it. For example, if a test shader verifies that sin(PI) == 0, the runtime optimizer will just turn that whole thing to "true", and never exercise the sin() function at all.

But of course, that approach means that the testsuite fails to test the optimizer itself, very important given how much aggressive runtime optimization we do.

So I made some changes to the CMake scripts to run each testsuite entry twice, once in the default way and another time with full optimization (this is communicated to testshade with a TESTSHADE_OPT environment variable that overrides command line arguments -- the alternative would be to change each of 72 testsuite/*/run.py files individually, and hope that all newly added ones will be done correctly, ick).

There were three tests where the output differed when optimized in expected ways, usually multi-layer shader groups where the optimizations figured out that the earlier layers didn't need to run at all, but then the printf functions in those layers no longer appeared in the test output. In those cases, I either checked in an alternate ref/out.opt.txt for the optimized output to match to, or changed the test slightly to make the optimized results match or be more helpful, or some combination.

There was also one test that revealed a legitimate bug! Which I fixed as part of this review -- we were allowing constants to be tagged as "needing derivs", which could happen if there was something like Dx(foo) and foo was folded down to a constant. Hilarity ensued, the reported derivs were not the zero as you'd expect from a real constant.