Silence a few -Wcast-align warnings in the headers

We don't use -Wcast-align ourselves, but the headers trigger warnings for those users who do, which is not nice.