Clean up endian-conversion code

This fixes issues on platforms that require byte-aligned reads and writes, such as ARMv5 and earlier.