BUG: Fix dtypes in sorting functions to npy_intp

Some of the iteration variables were int, causing sorts to fail for large
arrays. Fixes github issue 427.
BUG: Fix reporting of memory error inside sorting

PyDataMem_NEW was not checked for error and the occurance of
errors not returned correctly in new_sort. Also for PyArray_LexSort
and new_argsort it should now raise MemoryError correctly.
This is done by setting PyErr_NoMemory() when no error is already
present, as the low level sorting can only fail for this reason.
Merge pull request #2974 from seberg/issue-427

Fix some large array sorts and memory error handling