CxxTest/cxxtest — Issue #36: Random option to randomize test ordering
Author: whart222
Created: 2012-02-05 04:44:07+00:00
State: open

It could be useful to have an option in cxxtestgen.py that randomizes the order
that it runs the tests of a class. The randomization would probably happen only
when cxxtestgen runs, not actually at runtime for the test. But, this would
allow a build system itself to introduce randomness into the tests.

This might also go along with doing some new printers. For example, if
stdioprinter was used, and the fourth test fails because of some odd dependency
on the third test, then it would be useful to see which test was run in the
third slot (even though it succeeded).

------- Additional comments from Gašper Ažman Fri Sep 5 01:07:54 -0800 2008 -------

Actually, it's doable at run-time as well, and without hash maps.

In my previous job, I implemented a host of LFSR's - basically, they are n-bit
random number generators that are unique - they don't repeat numbers until they
wrap around. Fasther than mersenne twister. We needed the uniqueness aspect, and
so I already have the problem of running the tests in any order, but not
repeating, solved.

So, we just cram the test functions into an array (they all have the same sig
anyway) and use a usual-way seeded lfsr to produce the order. Easy as pie :)

If you're concerned about dependency on the 3rd test... I don't think that that
should be an issue. Test functions should be pure IMHO. Either that or you
shouldn't use the random runner. So, the printer is unnecessary. I agree it's a
good idea though.