BUG: add missing gufunc failure return values and missing malloc fail check

this is the path the gufunc nditer size overflow goes over causing the segfault later on as no error is set.
I'm not sure why it takes this path, probably it clobbers the ufunc somewhere.