Rand fixes

Proposed fixes for https://github.com/rubinius/rubinius/issues/3180

Does not completely fix case 3 as there is weird type coercion that goes deeper: https://github.com/rubinius/rubinius/issues/3181
The rule to not require standard lib in core lib specs is broken in a few places but those need to be fixed.

The code does not depend on BigDecimal and the specs should not either. The specs should describe the API and protocol (ie which methods are expected and which methods will be used to coerce values).