ENH: core: make unpickling with encoding='bytes' work

When loading Py2-generated pickles on Py3, it may be sometimes useful to use `pickle.load(..., encoding='bytes')` instead of `encoding='latin1'`.

This is currently blocked by `dtype.__setstate__` not accepting the endian as a byte string.
There's not really a good reason to not support this, so we could change it, as below.

The routine also seemed to have some old bug with missing error return, so fix that too.
would it be simpler to check the type of args[2] and change the format string accordingly instead of parsing twice?
PyArg_ParseTuple is a very slow function it could maybe slow down loading large pickles