math.floats: Fix the output of /mod and add some unit tests. Fixes #803.