ENH: core: make unpickling with encoding='bytes' work

When loading Py2-generated pickles on Py3, it may be sometimes useful to use `pickle.load(..., encoding='bytes')`.

This is currently blocked by `dtype.__setstate__` not accepting the endian as a byte string.
There's not really a good reason to not support this, so we could change it, as below.

The routine also seemed to have some old bug with missing error return, so fix that too.