crypto: Only throw if randomBytes arg is a non-number

If it is out of the acceptable range, then return a more sensible
error.  (Throw for sync, call the callback for async.)

Fixes GH-4583