Math: add test for buggy math library.

We rely on a mostly-working math library.  Warn users about trouble
we know how to detect.