Skip test_math unless using ta2

isfinite and friends require ta2 with musl headers.