This is by the way invalid on arm. It can cause unaligned reads for 32bit values, so we can't do this optim.