crypto: Only throw if randomBytes arg is a non-number

If it is out of the acceptable range, then return a more sensible error message.  (Throw for sync, call the callback for async.)

Implements the compromise suggested in #4583.