Hack to get `char` and `byte` to play well

This hack isn't great, but it fixes the problem. I don't like
being a problem-only kinda guy, so I'm submitting a patch.

Personally I think that cppformat shouldn't be so pedantic. It
seems like if there is a natural conversion from the source to the
destination format, then it should be performed.

Resolves #55