Clean up endian-conversion code

This fixes issues on platforms that require byte-aligned reads and writes,
such as ARMv5 and earlier.