Fix spec to actually use maximum uint value
Fix range check for rb_fix2uint

Fixes #2770