ENH: core: make unpickling with encoding='bytes' work

When loading Py2-generated pickles on Py3, it may be sometimes useful to use `pickle.load(..., encoding='bytes')` instead of `encoding='latin1'`.

This is currently blocked by `dtype.__setstate__` not accepting the endian as a byte string.
There's not really a good reason to not support this, so we could change it, as below.

The routine also seemed to have some old bug with missing error return, so fix that too.
actually shouldn't we just convert the endian object to ascii? anything else is not allowed as endian character anyway