Repository: numpy/numpy

Issue:
MaskedArray.mean returns zero length array when self._mask is nomask.  Should return scalar.
[Copied from this stackoverflow question](http://stackoverflow.com/questions/29689040/testing-equivalence-of-means-of-numpy-maskedarray-instances-raises-attribute-err):

**Edit:**

So far it appears that this is a bug that was added somewhere between versions 1.8.1 and 1.9.2.  I have yet to track down the root cause.  In 1.8.1 `np.ma.MaskedArray.mean()` would return a scalar.  In 1.9.2 it is returning a zero dimensional MaskedArray.  This appears to only occur when `foo.mask` is set to `np.ma.nomask`.

I tried to track this issue down, but am having difficulty following the inheritance for `MaskedArray.mean()`.  Specifically, I am having difficulty with line 2711 in numpy/core/fromnumeric.py which appears to call back to `MaskedArray.mean()` when called with a `MaskedArray` instance.

**Original post:**

I'm probably doing something funny here, but I can't seem to find the problem.  When I test the equivalence of the means of two different instances of `np.ma.MaskedArray` an `AttributeError` is raised.

**Creating the arrays:**

```
In [1]: import numpy as np

In [2]: foo = np.ma.array([1,2,3,4])

In [3]: bar = np.ma.array([1,2,3,4])

In [4]: foo.mean()
Out[4]: 
masked_array(data = 2.5,
             mask = False,
       fill_value = 1e+20)


In [5]: bar.mean()
Out[5]: 
masked_array(data = 2.5,
             mask = False,
       fill_value = 1e+20)
```

**Comparison of the arrays works fine:**    

```
In [6]: foo == bar
Out[6]: 
masked_array(data = [ True  True  True  True],
             mask = False,
       fill_value = True)
```

**Testing equivalence of the means fails:**

```
In [7]: foo.mean() == bar.mean()
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-7-3b824b0972e3> in <module>()
----> 1 foo.mean() == bar.mean()

/users/___/.local/lib/python2.7/site-packages/numpy/ma/core.pyc in __eq__(self, other)
   3705                         mask = np.all([[f[n].all() for n in mask.dtype.names]
   3706                                         for f in mask], axis=axis)
-> 3707                 check._mask = mask
   3708         return check
   3709     #

AttributeError: 'numpy.bool_' object has no attribute '_mask'

In [8]: foo.mean() != bar.mean()
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-8-0947fa5da1ed> in <module>()
----> 1 foo.mean() != bar.mean()

/users/___/.local/lib/python2.7/site-packages/numpy/ma/core.pyc in __ne__(self, other)
   3738                         mask = np.all([[f[n].all() for n in mask.dtype.names]
   3739                                         for f in mask], axis=axis)
-> 3740                 check._mask = mask
   3741         return check
   3742     #
```

**Testing for larger or smaller of the means works fine:**

```
AttributeError: 'numpy.bool_' object has no attribute '_mask'

In [9]: foo.mean() >= bar.mean()
Out[9]: 
masked_array(data = True,
             mask = False,
       fill_value = True)


In [10]: foo.mean() <= bar.mean()
Out[10]: 
masked_array(data = True,
             mask = False,
       fill_value = True)
```

**Python version 2.7.2 with Numpy version 1.9.2:**

```
In [11]: np.__version__
Out[11]: '1.9.2'
```

It appears that in `np.ma.MaskedArray.__ne__()` the variable `check` is created by a line that looks like:

```
check = np.ndarray.__eq__(foo.filled(0), bar.filled(0)).view(type(foo))
```

which I would expect to return a new masked array.  However, since `np.ndarray.__eq__()` returns an instance of `np.bool_`.  attempting to make a `MaskedArray` view of the `check` simply produces another `np.bool_` instance.  The error occurs when the routine attempts to assign to `check._mask` since `np.bool_` does not have a `_mask` attribute.

Any ideas what is going on here?  A bug or stupidity on my part?



Discussion:
I just came across the same bug. I think it's actually a bug in MaskedArray.array_wrap which I haven't figured out yet. 

The problem isn't in the line in `np.ma.MaskedArray.__ne__()` you sugested (as I also initially thought), because the problem occurs when self and other are 0-d arrays and the 0-d array case is supposed to be caught by the `if omask is nomask` or `if self is masked` cases a little above that. No, the problem is that omask.mask is wrong: If you do `m = np.ma.array([1,2,3]).mean()` and print m, it says,

```
masked_array(data = 2.0,
         mask = False,
   fill_value = 1e+20)
```

but looking more carefully, printing `m.mask` says `array(False, dtype=bool)` when it should really just say `False`. That causes the `if omask is nomask` test to fail. So far I've tracked down that the conversion of the mask  from `False`  to `array(False, dtype=bool)` happens in array wrap.

Any insight on what could be done to solve this issue ? It is quite annoying that the result of an operation depend on the method (median return a zero-length masked array whereas other methods return a scalar, #5969) or on the type of the mask (mean's result can be a scalar or a zero-length masked array if mask is nomask). 

``` python
In [1]: import numpy as np

In [2]: a = np.ma.arange(10, dtype=float)
__array_finalize__

In [3]: a
Out[3]:
masked_array(data = [ 0.  1.  2.  3.  4.  5.  6.  7.  8.  9.],
             mask = False,
       fill_value = 1e+20)

In [4]: b = np.ma.array(a, mask=False)
__array_finalize__

In [5]: b
Out[5]:
masked_array(data = [0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0],
             mask = [False False False False False False False False False False],
       fill_value = 1e+20)

In [6]: a.mean()
__array_wrap__
__array_finalize__
__array_wrap__
__array_finalize__
__array_finalize__
Out[6]:
masked_array(data = 4.5,
             mask = False,
       fill_value = 1e+20)

In [7]: b.mean()
Out[7]: 4.5

In [8]: a.sum()
Out[8]: 45.0

In [9]: b.sum()
Out[9]: 45.0

In [10]: np.ma.median(a)
__array_finalize__
Out[10]:
masked_array(data = 4.5,
             mask = False,
       fill_value = 1e+20)

In [11]: np.ma.median(b)
__array_finalize__
Out[11]:
masked_array(data = 4.5,
             mask = False,
```

I tried to look at the `__array_wrap__` and `__array_finalize__` methods, but it is a bit obscure to me ;).

Hmm, I can't check right now but I suspect this can be fixed by #7350. 

It may also be fixed by #5706, since I totally rewrite `MaskedArray.mean` there.

Just a quick code snippet highlighting how I stumbled upon this issue with NumPy version 1.11.1, that may or may not provide further insight.

``` python
import numpy as np
x = np.linspace(0, 10, 11)  # dtype = float64
is_bad_data = x < 0.0  # no bad data
print("The mean of x: {:.2f}".format(np.ma.masked_where(is_bad_data, x).mean()))
```

 Which returns **ValueError: Unknown format code 'f' for object of type 'str'**.

Hmm, it looks like #7350 and #5706 happened to partially fixed this issue, but we still need to convert 0-d arrays to scalars when `mask == nomask`. Now that #7350 and #5706 are in it should be an easy fix, let me see.


Resolving patch:
diff --git a/numpy/ma/core.py b/numpy/ma/core.py
--- a/numpy/ma/core.py
+++ b/numpy/ma/core.py
@@ -5057,7 +5057,7 @@ def mean(self, axis=None, dtype=None, out=None, keepdims=np._NoValue):
 
         if self._mask is nomask:
             result = super(MaskedArray, self).mean(axis=axis,
-                                                   dtype=dtype, **kwargs)
+                                                   dtype=dtype, **kwargs)[()]
         else:
             dsum = self.sum(axis=axis, dtype=dtype, **kwargs)
             cnt = self.count(axis=axis, **kwargs)
@@ -5134,8 +5134,14 @@ def var(self, axis=None, dtype=None, out=None, ddof=0,
 
         # Easy case: nomask, business as usual
         if self._mask is nomask:
-            return self._data.var(axis=axis, dtype=dtype, out=out,
-                                  ddof=ddof, **kwargs)
+            ret = super(MaskedArray, self).var(axis=axis, dtype=dtype, out=out,
+                                               ddof=ddof, **kwargs)[()]
+            if out is not None:
+                if isinstance(out, MaskedArray):
+                    out.__setmask__(nomask)
+                return out
+            return ret
+
         # Some data are masked, yay!
         cnt = self.count(axis=axis, **kwargs) - ddof
         danom = self - self.mean(axis, dtype, keepdims=True)


Test patch:
diff --git a/numpy/ma/tests/test_core.py b/numpy/ma/tests/test_core.py
--- a/numpy/ma/tests/test_core.py
+++ b/numpy/ma/tests/test_core.py
@@ -3340,6 +3340,22 @@ def test_dot_shape_mismatch(self):
         assert_almost_equal(z.filled(0), [[1, 0], [15, 16]])
         assert_almost_equal(z.mask, [[0, 1], [0, 0]])
 
+    def test_varmean_nomask(self):
+        # gh-5769
+        foo = array([1,2,3,4], dtype='f8')
+        bar = array([1,2,3,4], dtype='f8')
+        assert_equal(type(foo.mean()), np.float64)
+        assert_equal(type(foo.var()), np.float64)
+        assert((foo.mean() == bar.mean()) is np.bool_(True))
+
+        # check array type is preserved and out works
+        foo = array(np.arange(16).reshape((4,4)), dtype='f8')
+        bar = empty(4, dtype='f4')
+        assert_equal(type(foo.mean(axis=1)), MaskedArray)
+        assert_equal(type(foo.var(axis=1)), MaskedArray)
+        assert_(foo.mean(axis=1, out=bar) is bar)
+        assert_(foo.var(axis=1, out=bar) is bar)
+
     def test_varstd(self):
         # Tests var & std on MaskedArrays.
         (x, X, XX, m, mx, mX, mXX, m2x, m2X, m2XX) = self.d