Difficulty should be unconditionally byteswapped, not swapped to big endian.