Rand fixes

Proposed fixes for https://github.com/rubinius/rubinius/issues/3180

Does not completely fix case 3 as there is weird type coercion that goes deeper: https://github.com/rubinius/rubinius/issues/3181
This description should properly mention that the range consists out of BigDecimal objects and not just regular numeric values.