Fix some large array sorts and memory error handling

This is a simple fix for large array sorts failing due to using int instead of npy_intp. It also fixes and changes error handling slightly because (as far as I can tell right now) one cannot set an error while the GIL is released.

While the fix itself is very simple, it is unfortunately not possible to do real tests for either of those (the memory error is tested for one case). The other would (once again I guess) require a high mem (and very slow) test decorator. Though here it is impossible to break things by using npy_intp instead of int.

This closes Issue gh-427