Repository: numpy/numpy
Pull request: MAINT:  better MemoryError message (#13225)
 I am really not sure if this is the best place to place this message and if it is the good call to suppress other debug info. 

Closes #13225.
<!-- Please be sure you are following the instructions in the dev guidelines
http://www.numpy.org/devdocs/dev/gitwash/development_workflow.html
-->
<!-- We'd appreciate it if your commit message is properly formatted
http://www.numpy.org/devdocs/dev/gitwash/development_workflow.html#writing-the-commit-message
-->


Problem statement:
Indicate attempted allocation size when constructing an array gives a MemoryError
I've seen multiple questions now of the form "why does this give a MemoryError" when the reason is the user ended up allocating a bigger array than expected.

It would be helpful to include the shape and dtype in the error message.

Hints from issue discussion:
I will give this a try sometimes this week. If someone else is already working on it let me know. 
For some reason this does not reproduce as reported on my setup (OSx, python 3.7.3): 

```
>>> import numpy as np
>>> t = np.zeros((51,765,285),dtype=np.float64)
>>> z = np.zeros((51,765,285),dtype=np.float64)
>>> indices = np.apply_along_axis(lambda a: a.searchsorted(10), axis=0, arr=t)
>>> indices.shape
(765, 285)
>>> np.take(z, indices, axis=0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/bma/numpy/numpy/core/overrides.py", line 150, in public_api
    implementation, public_api, relevant_args, args, kwargs)
  File "/Users/bma/numpy/numpy/core/fromnumeric.py", line 192, in take
    return _wrapfunc(a, 'take', indices, axis=axis, out=out, mode=mode)
  File "/Users/bma/numpy/numpy/core/fromnumeric.py", line 60, in _wrapfunc
    return bound(*args, **kwds)
IndexError: index 51 is out of bounds for size 51
>>> z[indices,:,:]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndexError: index 51 is out of bounds for axis 0 with size 51
```

On the other hand, a fresh install of numpy from git on Elementary OS reproduced it (both setups run 1.17.0). 
I'll keep looking.
@bmakos: You could try testing with `np.ones((2**40,), np.uint8)`, which should be easily greater than your available ram
<img width="1050" alt="numpy extensive memory use" src="https://user-images.githubusercontent.com/5707813/55749327-f10bcd00-5a40-11e9-927c-1e3b21739404.png">

Ok, so it seems I will not be able to reproduce this on my machine, the process gets automatically killed at around 120gb by the os. I'll just use a vm with Elemenary for this issue. Sry for the confusion. 
I am a bit insecure about the solution I took. Is it acceptable to issue just this amount of information for this error? 
These lines are where the change would go:

https://github.com/numpy/numpy/blob/f4a24b1f4779d1b9966efe2ad79cba21241ce500/numpy/core/src/multiarray/ctors.c#L1054-L1080
Yes, I was already working on adding it. I debugged it with gdb last night and I got to the same conclusion: 

```
Breakpoint 1, 0x0000555555602337 in PyErr_NoMemory ()
(gdb) up
#1  0x00007ffff5b3b131 in PyArray_NewFromDescr_int (subtype=0x7ffff60b9d00 <PyArray_Type>,
    descr=0x7ffff60ba660 <DOUBLE_Descr>, nd=4, dims=0x7fffffffcb30, strides=0x0, data=0x0, flags=0,
    obj=0x7ffff6163ad0, base=0x0, zeroed=0, allow_emptystring=0)
    at numpy/core/src/multiarray/ctors.c:1075
1075                PyErr_NoMemory();
(gdb)
```
I'll use `PyErr_SetObject` or `PyErr_SetString`, I just have to figure out how to build the message. 

@eric-wieser Thank you for confirming the location, it's good to know I am on track. 
One trick you could use is to dispatch the message formatting to python. One approach would be to add a helper function to `_internal.py`, the other would be to create a MemoryError subclass in `_exceptions.py`. search the c code for the pyhon function names found in those files.
@eric-wieser In the end I went with not using python to format the message because I struggled with passing the dimensions array to python and all the data I needed was available in `fa`. Also, found some handy helper functions (like `PyUString_FromFormat`). 
If you still wish to go the route you suggested I can probably do it, let me know. 
> I struggled with passing the dimensions array to python

You can use `PyArray_IntTupleFromIntp` to get the tuple of dimensions

Commit message: ENH: improve memory error message by adding shape and data type (#13225)

Patch to review:
diff --git a/numpy/core/_exceptions.py b/numpy/core/_exceptions.py
index a1997ef6b900..3af006ca39dd 100644
--- a/numpy/core/_exceptions.py
+++ b/numpy/core/_exceptions.py
@@ -121,3 +121,17 @@ def __init__(self, axis, ndim=None, msg_prefix=None):
                 msg = "{}: {}".format(msg_prefix, msg)
 
         super(AxisError, self).__init__(msg)
+
+
+@_display_as_base
+class _ArrayMemoryError(MemoryError):
+    """ Thrown when an array cannot be allocated"""
+    def __init__(self, shape, dtype):
+        self.shape = shape
+        self.dtype = dtype
+
+    def __str__(self):
+        return "Attempted to allocate array with shape " + \
+        str(self.shape).rstrip(')').rstrip(',') + ") and data type " + \
+        str(self.dtype) + "."
+
diff --git a/numpy/core/src/multiarray/ctors.c b/numpy/core/src/multiarray/ctors.c
index 42cd31069b39..9677a13d915a 100644
--- a/numpy/core/src/multiarray/ctors.c
+++ b/numpy/core/src/multiarray/ctors.c
@@ -1072,8 +1072,30 @@ PyArray_NewFromDescr_int(PyTypeObject *subtype, PyArray_Descr *descr, int nd,
             data = npy_alloc_cache(nbytes);
         }
         if (data == NULL) {
-            PyErr_NoMemory();
-            goto fail;
+            static PyObject *exc_type = NULL;
+            
+            npy_cache_import(
+                "numpy.core._exceptions", "_ArrayMemoryError",
+                &exc_type);
+            if (exc_type == NULL) {
+                return NULL;
+            }
+            
+            PyObject *shape = PyArray_IntTupleFromIntp(fa->nd,fa->dimensions);
+            if (shape == NULL) {
+                return NULL;
+            }
+            
+            /* produce an error object */
+            PyObject *exc_value = PyTuple_Pack(2, shape, descr);
+            Py_DECREF(shape);
+            if (exc_value == NULL){
+                return NULL;
+            }
+            PyErr_SetObject(exc_type, exc_value);
+            Py_DECREF(exc_value);
+            return NULL;
+
         }
         fa->flags |= NPY_ARRAY_OWNDATA;
 


Code review comments:
diff_hunk: @@ -121,3 +121,17 @@ def __init__(self, axis, ndim=None, msg_prefix=None):
                 msg = "{}: {}".format(msg_prefix, msg)
 
         super(AxisError, self).__init__(msg)
+
+
+@_display_as_base
+class _ArrayMemoryError(MemoryError):
+    """ Thrown when an array cannot be allocated"""
+    def __init__(self, shape, dtype):
+        self.shape = shape
+        self.dtype = dtype
+
+    def __str__(self):
+        return "Attempted to allocate array with shape " + \
+        str(self.shape).rstrip(')').rstrip(',') + ") and data type " + \
+        str(self.dtype) + "."
path: numpy/core/_exceptions.py
text: @user1:
I don't think you should remove the trailing comma from the `shape`.

Also, consider using `.format` here, as
```python
return "Unable to allocate array with shape {} and data type {}".format(self.shape, self.dtype)
```