BUG: Fix dtypes in sorting functions to npy_intp

Some of the iteration variables were int, causing sorts to fail for large
arrays. Fixes github issue 427.
BUG: Fix reporting of memory error inside sorting

PyDataMem_NEW was not checked for error and the occurance of
errors not returned correctly in new_sort. Also for PyArray_LexSort
and new_argsort it should now raise MemoryError correctly.
This is done by setting PyErr_NoMemory() when no error is already
present, as the low level sorting can only fail for this reason.
TST: Test memory error during sort

Unfortunatly it is not possible to test all of the possible errors
individually, or test argsort/lexsort easily, since it is difficult
to adjust when exactly one runs out of memory (short of the first
occurance)