petdance/scratch — Issue #21: Enable meaningful testing of t/native_pbc/*.t
Author: petdance
Created: 2011-11-01 05:29:36+00:00
State: open
Labels: cage, Pri-normal, Sev-medium, testing

Version 0.9.1 was released with failing t/native_pbc/*.t tests. If the tests are to be included in 1.0, it would be nice to have confidence that they will pass in the released version.

Offhand, I can think of three big problems that contributed to the failure. I hasten to add that *all* of these were out of the control of the release manager -- they are inherent in the design of the tests.

1. The tests act differently in "DEVELOPING" versus released directories, meaning all the tests done in an svn-checkout -- even those done just prior to release -- weren't necessarily relevant.

2. The release procedure is difficult in practice. It is not easy, on short notice, to get developers with all the requisite different platforms to run the appropriate scripts, commit the appropriate files, and re-run all the appropriate tests. Also, because of item 1, even running all the appropriate tests does not ensure that the tests will actually pass in the released version.

3. There was no meaningful code freeze prior to the release, making it essentially impossible to do the coordination necessary for item 2.

What to do?

First, if the tests are to be kept, they must be designed so that they can be meaningfully run well in advance of the release. That way, new raw pbc files can be generated ahead of time, committed, and tested, and failures can be addressed. If they cannot be redesigned this way, then I think they are too fragile to waste time on, and they should simply be deleted.

Second, I think the ideas of having a code freeze and issuing release candidates merit serious consideration prior to 1.0.
Trac ID #357