astropy/astropy — pull request #2087 by embray (2014-02-12 23:52:34+00:00)
Title: Parse and do float comparison of float output in doctests

This is a suggestion I've made in a few places but I don't think I ever really made a ticket for it, so here it is: Many issues with doctests arise when slight changes in numerical code between Astropy versions, Python versions, or worst of all platforms result in slight differences in floating point results, causing the doctests to fail, since they work entirely on string comparison. This is even worse because on different platforms the default string formatting for different floating point values can differ.

Most of this time we address this using ellipses (`...` ) to ignore values above a certain decimal place, a feature supported by doctest. But this does not even work for all cases--for example I recently had a case where the doctest expected something like `1.000000000000001e-23`, but my system was rounding this to `1e-23`. I suppose this could be written `1...e-23` in the doctest, but that's pretty unclear.

What if instead the doctest runner could actually parse floating point strings into actual float values and do a direct numerical comparison? Optionally, the comparison could even be made with a specific tolerance.

This could be cumbersome on tests that don't expect floating point values in the output, so I would suggest this be doing with a new doctest directive, perhaps `# doctest: +FLOAT`. This would use a regular expression to parse out all floating point values found in expected and actual test output, in order, so that they can be matched up one for one and compared. This flexible approach would allow it to work with floating point values that appear anywhere in the output, such as in the `repr` for a `Quantity`.