Rand fixes

Proposed fixes for https://github.com/rubinius/rubinius/issues/3180

Does not completely fix case 3 as there is weird type coercion that goes deeper: https://github.com/rubinius/rubinius/issues/3181
Unfortunately, Ruby does not have a built-in float-like object. Moving this test into BigDecimal would no longer be testing the actual implementation.