Ensure oversized types don't trigger unsafety

Should fix #17913.

Also clean-up u64/u32-ness. I really should split this commit and add tests (I have no idea how to add them).