add checked_{add,sub,mul} for BigInt
fix #5935 (rationalize returns curious/incorrect values)

Partially addresses #7501.
Merge pull request #8216 from JuliaLang/mn/rationalize

fix rationalize (#5935)