ENH: core: make unpickling with encoding='bytes' work

Make dtype.__setstate__ accept endian as a byte string on Python 3.
BUG: core: fix missing error return

Introduced in c3551579, apparently mistake.