[RFC] Fix some more -Wconversion warnings.

This is on top of #1488, but I'll rebase as soon as that one is merged.
:+1: