Remove ENDIAN conditional compilation

The compiler should automatically generate the same code, so no need for this mess anymore.