Cleanup serialize

This brings over some changes from [rustc-serialize](https://github.com/rust-lang/rustc-serialize). It makes sense to keep the two in sync until we finally remove libserialize, just to make sure they don't diverge from each other.
already done in #19974