Ensure oversized types don't trigger unsafety

Also clean-up u64/u32-ness. I really should split this commit.