Repository: numpy/numpy

Issue:
BUG: MaskedArray with nested dtype and object elements cause AttributeError on access
When I create a `MaskedArray` using a nested dtype which contains an `object` at the innermost level, trying to access it causes an _AttributeError: 'str' object has no attribute 'ndim'_.

If I use `int` instead of `object`, the element access works fine.

The issue seems to be caused by the assumption that `_fill_value` is of a numpy type and has an `ndim` attribute.
This assumption is true for `numpy.int32` which would be used if `int` was specified inside the dtype.
But for `object`, the `_fill_value` will be a just python `str` _'?'_, which has no attribute `ndim`.

I propose to change the code line in question to
```python
if hasattr(dout._fill_value, 'ndim') and dout._fill_value.ndim > 0:
```

### Reproducing code example:

```python
import numpy as np
my_dtype = np.dtype([('b', [('c', object)], (1,))])
a = np.ma.masked_all((1,), my_dtype)
print(a['b']['c'])
```

### Error message:

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python37\lib\site-packages\numpy\ma\core.py", line 3276, in __getitem__
    if dout._fill_value.ndim > 0:
AttributeError: 'str' object has no attribute 'ndim'
```

### Numpy/Python version information:

1.18.1 3.7.4 (tags/v3.7.4:e09359112e, Jul  8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)]




Discussion:
Nice catch

but shouldn't we just check if __fill_value is not a ```str``` rather than use hasattr. But even then it feels hacky, there seems to be a comment regarding issue #6723 above the check. 

is this the right approach @seberg ? Trying to understand the background better.
Well, its a bit of a mess. But aside from that, you are right of course. Checking for `ndim` has a bad code smell (and could definitely fail). Even the current code _can_ do something incorrect for certain `object` cases in theory.
I think the solution has to be to test for this happening more directly, one thing I am considering is changing it along:
```
new_fill_value = self._fill_value[indx, ...]  # guarantee array return
```
and moving the `flat` line out of the if. Or, alternative adding an `else: dout._fill_value = new_fill_value[()]`.
So, the purpose of `_fill_value` was to _always_ be an array, unlike `.fill_value` which is the scalar.

Do we know how `_fill_value` is ending up with the wrong type?
Ah sorry, this is of course structured access, which is the problem (so the `...` suggestion will not work on its own). In any case, this code branch seems to not know about `_fill_value` being guaranteed to be an array, and it has to make sure of that.

Probably the best thing to do is to get the new dtype directly with `dtype = self._fill_value.dtype[indx]`, and then using `dtype.ndim` to check for that special branch and otherwise create a new empty array with that dtype and assign to it to ensure that the new `_fill_value` is again an array.
I have looked into the issue, a little more

I have a better understanding of masked array

The solution would be something like this? @seberg 
```python
                    if self._fill_value.dtype[indx].ndim > 0:
                        if not (dout._fill_value ==
                                dout._fill_value.flat[0]).all():
                            warnings.warn(
                                "Upon accessing multidimensional field "
                                f"{indx!s}, need to keep dimensionality "
                                "of fill_value at 0. Discarding "
                                "heterogeneous fill_value and setting "
                                f"all to {dout._fill_value[0]!s}.",
                                stacklevel=2)
                        dout._fill_value = dout._fill_value.flat[0]
                    else:
                        dout._fill_value = array(self._fill_value[indx])
```

Resolving patch:
diff --git a/numpy/ma/core.py b/numpy/ma/core.py
--- a/numpy/ma/core.py
+++ b/numpy/ma/core.py
@@ -3297,6 +3297,10 @@ def _scalar_heuristic(arr, elem):
                 if self._fill_value is not None:
                     dout._fill_value = self._fill_value[indx]
 
+                    # Something like gh-15895 has happened if this check fails.
+                    # _fill_value should always be an ndarray.
+                    if not isinstance(dout._fill_value, np.ndarray):
+                        raise RuntimeError('Internal NumPy error.')
                     # If we're indexing a multidimensional field in a
                     # structured array (such as dtype("(2,)i2,(2,)i1")),
                     # dimensionality goes up (M[field].ndim == M.ndim +
@@ -3304,7 +3308,7 @@ def _scalar_heuristic(arr, elem):
                     # M[field] but problematic for M[field].fill_value
                     # which should have shape () to avoid breaking several
                     # methods. There is no great way out, so set to
-                    # first element.  See issue #6723.
+                    # first element. See issue #6723.
                     if dout._fill_value.ndim > 0:
                         if not (dout._fill_value ==
                                 dout._fill_value.flat[0]).all():
@@ -3315,7 +3319,10 @@ def _scalar_heuristic(arr, elem):
                                 "heterogeneous fill_value and setting "
                                 f"all to {dout._fill_value[0]!s}.",
                                 stacklevel=2)
-                        dout._fill_value = dout._fill_value.flat[0]
+                        # Need to use `.flat[0:1].squeeze(...)` instead of just
+                        # `.flat[0]` to ensure the result is a 0d array and not
+                        # a scalar.
+                        dout._fill_value = dout._fill_value.flat[0:1].squeeze(axis=0)
                 dout._isfield = True
             # Update the mask if needed
             if mout is not nomask:


Test patch:
diff --git a/numpy/ma/tests/test_extras.py b/numpy/ma/tests/test_extras.py
--- a/numpy/ma/tests/test_extras.py
+++ b/numpy/ma/tests/test_extras.py
@@ -64,6 +64,28 @@ def test_masked_all(self):
         control = array([[(1, (1, 1))]], mask=[[(1, (1, 1))]], dtype=dt)
         assert_equal(test, control)
 
+    def test_masked_all_with_object_nested(self):
+        # Test masked_all works with nested array with dtype of an 'object'
+        # refers to issue #15895
+        my_dtype = np.dtype([('b', ([('c', object)], (1,)))])
+        masked_arr = np.ma.masked_all((1,), my_dtype)
+
+        assert_equal(type(masked_arr['b']), np.ma.core.MaskedArray)
+        assert_equal(type(masked_arr['b']['c']), np.ma.core.MaskedArray)
+        assert_equal(len(masked_arr['b']['c']), 1)
+        assert_equal(masked_arr['b']['c'].shape, (1, 1))
+        assert_equal(masked_arr['b']['c']._fill_value.shape, ())
+    
+    def test_masked_all_with_object(self):
+        # same as above except that the array is not nested
+        my_dtype = np.dtype([('b', (object, (1,)))])
+        masked_arr = np.ma.masked_all((1,), my_dtype)
+
+        assert_equal(type(masked_arr['b']), np.ma.core.MaskedArray)
+        assert_equal(len(masked_arr['b']), 1)
+        assert_equal(masked_arr['b'].shape, (1, 1))
+        assert_equal(masked_arr['b']._fill_value.shape, ())
+
     def test_masked_all_like(self):
         # Tests masked_all
         # Standard dtype