random.c: coerce before check negative

* random.c (rb_random_ulong_limited): coerce before check negative.
  [Fixes GH-379]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@42542 b2dd03c8-39d4-4d8f-98ff-823fe69b080e