Repository: numpy/numpy

Issue:
Problem with views on masked structured arrays
Create a view on a structured array (e.g. selecting a list of columns) causes problems with comparisons when the array is masked.

### Example ###

Create a view on a masked structured array:
```python
import numpy as np
data = np.ma.array([
  ('Mon', 'Sunny',  30, 20),
  ('Tue', 'Cloudy', 28, 17),
  ('Wed', 'Rain',   25, 17),
  ('Thu', 'Sunny',  29, 18),
  ('Fri', 'Sunny',  30, 20),
], dtype=[('day','|S3'),('weather','|S10'),('high','i4'),('low','i4')])
data2 = data[['day','high']]
print (data2)
```
```
[('Mon', 30) ('Tue', 28) ('Wed', 25) ('Thu', 29) ('Fri', 30)]
```
Try to do a comparison on the array:
```python
print (data2[1:] == data2[:-1])
```
```
Traceback (most recent call last):
  File "test_numpy14.py", line 12, in <module>
    print (data2[1:] == data2[:-1])
  File "XXX/lib/python2.7/site-packages/numpy/ma/core.py", line 4033, in __eq__
    return self._comparison(other, operator.eq)
  File "XXX/lib/python2.7/site-packages/numpy/ma/core.py", line 3993, in _comparison
    sdata = sbroadcast.filled(odata)
  File "XXX/lib/python2.7/site-packages/numpy/ma/core.py", line 3709, in filled
    fill_value = _check_fill_value(fill_value, self.dtype)
  File "XXX/lib/python2.7/site-packages/numpy/ma/core.py", line 459, in _check_fill_value
    raise ValueError(err_msg % (fill_value, fdtype))
ValueError: Unable to transform [('Mon', 30) ('Tue', 28) ('Wed', 25) ('Thu', 29)] to dtype [('day', '|S3'), ('', '|V10'), ('high', '<i4'), ('', '|V4')]
```
A similar problem happens when calling `numpy.ma.unique` on the view.  The problem persists even after copying the data (e.g. `data2 = data2.copy()`)

I am using numpy 1.14.0 with Python 2.7.6.  The problem does not appear in numpy 1.13.3, or when using a regular (non-masked) structured array.


Discussion:
Thanks for the report, confirmed.

This is very similar to, or caused by, #10387, which itself is indirectly caused by our recent change in numpy 1.14.0 to return a view instead of a copy for multi-field indexing.

We're on it. We are planning to revert that change in 1.14.1, and push it off to 1.15. That will give us time to fix this and similar bugs that need to be fixed first (#3176, #8100).

This report is a little different from #10387, because it is likely that the `numpy.ma` code should not be using `dtype.descr` in the first place. So in addition to fixing the padding bytes in `.descr`, we should also remove use of `.descr` in `np.ma`.


`fill_value` has some really weird semantics that are far too complex, when the logic should be nothing more than `self.fill_value = np.array(fill_value, dtype=self.dtype)`. Of course, that will certainly break compatibility.

Perhaps we should start deprecating cases that give a different result.
Pushing off to 1.16. @ahaldane How do you want to handle the 1.15 fix? If you want to keep master as is we should wait until 1.15 is branched.

Resolving patch:
diff --git a/numpy/ma/core.py b/numpy/ma/core.py
--- a/numpy/ma/core.py
+++ b/numpy/ma/core.py
@@ -449,17 +449,15 @@ def _check_fill_value(fill_value, ndtype):
 
     """
     ndtype = np.dtype(ndtype)
-    fields = ndtype.fields
     if fill_value is None:
         fill_value = default_fill_value(ndtype)
-    elif fields:
-        fdtype = [(_[0], _[1]) for _ in ndtype.descr]
+    elif ndtype.names is not None:
         if isinstance(fill_value, (ndarray, np.void)):
             try:
-                fill_value = np.array(fill_value, copy=False, dtype=fdtype)
+                fill_value = np.array(fill_value, copy=False, dtype=ndtype)
             except ValueError:
                 err_msg = "Unable to transform %s to dtype %s"
-                raise ValueError(err_msg % (fill_value, fdtype))
+                raise ValueError(err_msg % (fill_value, ndtype))
         else:
             fill_value = np.asarray(fill_value, dtype=object)
             fill_value = np.array(_recursive_set_fill_value(fill_value, ndtype),
@@ -780,6 +778,10 @@ def fix_invalid(a, mask=nomask, copy=True, fill_value=None):
     a._data[invalid] = fill_value
     return a
 
+def is_string_or_list_of_strings(val):
+    return (isinstance(val, basestring) or
+            (isinstance(val, list) and
+             builtins.all(isinstance(s, basestring) for s in val)))
 
 ###############################################################################
 #                                  Ufuncs                                     #
@@ -3245,7 +3247,7 @@ def _scalar_heuristic(arr, elem):
             # Inherit attributes from self
             dout._update_from(self)
             # Check the fill_value
-            if isinstance(indx, basestring):
+            if is_string_or_list_of_strings(indx):
                 if self._fill_value is not None:
                     dout._fill_value = self._fill_value[indx]
 


Test patch:
diff --git a/numpy/ma/tests/test_core.py b/numpy/ma/tests/test_core.py
--- a/numpy/ma/tests/test_core.py
+++ b/numpy/ma/tests/test_core.py
@@ -2029,6 +2029,17 @@ def test_fillvalue(self):
         assert_equal(x.fill_value, 999.)
         assert_equal(x._fill_value, np.array(999.))
 
+    def test_subarray_fillvalue(self):
+        # gh-10483   test multi-field index fill value
+        fields = array([(1, 1, 1)],
+                      dtype=[('i', int), ('s', '|S8'), ('f', float)])
+        with suppress_warnings() as sup:
+            sup.filter(FutureWarning, "Numpy has detected")
+            subfields = fields[['i', 'f']]
+            assert_equal(tuple(subfields.fill_value), (999999, 1.e+20))
+            # test comparison does not raise:
+            subfields[1:] == subfields[:-1]
+
     def test_fillvalue_exotic_dtype(self):
         # Tests yet more exotic flexible dtypes
         _check_fill_value = np.ma.core._check_fill_value