fix hashing of rationals (#9264)

Fixes #9264, also replace `x*2.0^y` by faster `ldexp(x,y)`.
Couldn't this just be `h = hash_uint((n % UInt) $ h) $ h`?