[RFC] Fix some more -Wconversion warnings.

This is on top of #1488, but I'll rebase as soon as that one is merged (if this is even necessary).