Rand fixes

Proposed fixes for https://github.com/rubinius/rubinius/issues/3180

Does not completely fix case 3 as there is weird type coercion that goes deeper: https://github.com/rubinius/rubinius/issues/3181
@brixen What would be the best way of testing float-like numbers in core? These bugs are happening because the code eagerly checks for integer-ness instead of float-ness, but I can't reproduce them without a separate Float implementation.

Also in general, is there a good way of detecting integer-ness vs float-ness? Seems like MRI doesn't do a good job (rational often behaves like integers).