Repository: numpy/numpy

Issue:
Do not warn when invalid values are masked in a MaskedArray
When doing a mathematical operation on a masked array, and some elements are invalid for the particular mathematical operation, numpy should not issue a warning if all invalid elements are masked.

This cannot be handled by `numpy.seterr`: I want numpy to warn or raise when performing an invalid operation on any non-masked value, but to be silent when performing an invalid operation on a masked value.

```
In [1]: A = arange(-2, 5)/2

In [2]: Am = numpy.ma.masked_less_equal(A, 0)

In [3]: numpy.log10(Am)
/export/data/home/gholl/venv/gerrit/bin/ipython3:1: RuntimeWarning: divide by zero encountered in log10
  #!/export/data/home/gholl/venv/gerrit/bin/python3.4
/export/data/home/gholl/venv/gerrit/bin/ipython3:1: RuntimeWarning: invalid value encountered in log10
  #!/export/data/home/gholl/venv/gerrit/bin/python3.4
Out[3]: 
masked_array(data = [-- -- -- -0.3010299956639812 0.0 0.17609125905568124 0.3010299956639812],
             mask = [ True  True  True False False False False],
       fill_value = 1e+20)
```

BUG: MaskedArray does not seem to respect ufunc dispatch hierarchy
<!-- Please describe the issue in detail here, and fill in the fields below -->
When using a [custom array container](https://docs.scipy.org/doc/numpy-1.17.0/user/basics.dispatch.html) that implements `__array_ufunc__` and is meant to be able to wrap MaskedArrays, non-commutativity occurs in binary operations with MaskedArrays. Despite what the below comment mentions:

https://github.com/numpy/numpy/blob/v1.17.3/numpy/ma/core.py#L3960-L3972

it does not [properly defer](https://numpy.org/neps/nep-0013-ufunc-overrides.html#type-casting-hierarchy), as seen in the example below. I am unfortunately not well-acquainted with the MaskedArray internals, so I don't know what would be a good way forward for a fix.

xref https://github.com/hgrecco/pint/issues/633

### Reproducing code example:

<!-- A short code example that reproduces the problem/missing feature. It should be
self-contained, i.e., possible to run as-is via 'python myproblem.py' -->

```python
import numpy as np
import numpy.lib.mixins


class WrappedArray(numpy.lib.mixins.NDArrayOperatorsMixin):
    __array_priority__ = 20

    def __init__(self, array, **attrs):
        self._array = array
        self.attrs = attrs

    def __repr__(self):
        return f"{self.__class__.__name__}(\n{self._array}\n{self.attrs}\n)"

    def __array__(self):
        return np.asarray(self._array)

    def __array_ufunc__(self, ufunc, method, *inputs, **kwargs):
        if method == '__call__':
            inputs = [arg._array if isinstance(arg, self.__class__) else arg
                      for arg in inputs]
            return self.__class__(ufunc(*inputs, **kwargs), **self.attrs)
        else:
            return NotImplemented


# Show basic wrap
w = WrappedArray(np.arange(3), test=1)
print(w)
print()

# Wrapping MaskedArrays works fine
wm = WrappedArray(np.ma.masked_array([1, 3, 5], mask=[False, True, False]),
                  test=2)
print(wm)
print()

# Operations with ndarrays work fine
a = np.array([2, 0, 1])
print(a * w)
print()

# Operations with masked arrays are not commutative
m = np.ma.masked_array([2, 0, 1], mask=[False, True, False])

# Good
print(w * m)
print()

# Bad
print(m * w)
```

Output:

```
WrappedArray(
[0 1 2]
{'test': 1}
)

WrappedArray(
[1 -- 5]
{'test': 2}
)

WrappedArray(
[0 0 2]
{'test': 1}
)

WrappedArray(
[0 -- 2]
{'test': 1}
)

[0 -- 2]
```

<!-- Remove these sections for a feature request -->

### Numpy/Python version information:

<!-- Output from 'import sys, numpy; print(numpy.__version__, sys.version)' -->

1.17.3 3.6.7 | packaged by conda-forge | (default, Nov  6 2019, 16:19:42) 
[GCC 7.3.0]



Discussion:
This could probably be fixed using a combination of `__numpy_ufunc__` and
`where=`, if anyone wants to try.

On Tue, Aug 12, 2014 at 8:22 PM, gerritholl notifications@github.com
wrote:

> When doing a mathematical operation on a masked array, and some elements
> are invalid for the particular mathematical operation, numpy should not
> issue a warning if all invalid elements are masked.
> 
> This cannot be handled by numpy.seterr: I want numpy to warn or raise
> when performing an invalid operation on any non-masked value, but to be
> silent when performing an invalid operation on a masked value.
> 
> In [1]: A = arange(-2, 5)/2
> 
> In [2]: Am = numpy.ma.masked_less_equal(A, 0)
> 
> In [3]: numpy.log10(Am)
> /export/data/home/gholl/venv/gerrit/bin/ipython3:1: RuntimeWarning: divide by zero encountered in log10
>   #!/export/data/home/gholl/venv/gerrit/bin/python3.4
> /export/data/home/gholl/venv/gerrit/bin/ipython3:1: RuntimeWarning: invalid value encountered in log10
>   #!/export/data/home/gholl/venv/gerrit/bin/python3.4
> Out[3]:
> masked_array(data = [-- -- -- -0.3010299956639812 0.0 0.17609125905568124 0.3010299956639812],
>              mask = [ True  True  True False False False False],
>        fill_value = 1e+20)
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/numpy/numpy/issues/4959.

## 

Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org

A little experimentation showed that np.ma.log10() worked on the example above. Is the intention for the default math functions to call the masked array function? 

In a closely related issue, supplying nan values in masked data causes an error in the safe domain calculations. Here the array's `data` is used in comparison functions even if it is masked. For example `np.array([np.nan]) > np.array([4])` should and does raise an error.

The test system is Debian 7.8 with the latest numpy 1.9.2 installed in ~/.local
(The same behaviour is obtained from WinPython 2.7.9, numpy 1.9.2)

## Test code

``` python
import sys
import numpy as np

print(sys.version)
print("np.__version__ {0}".format(np.__version__))

A = np.arange(-2, 5)/2
Am = np.ma.masked_less_equal(A, 0)
print('np.ma.log10(Am) works OK.') 
tm = np.ma.log10(Am)
print(tm.data)
print(tm.mask)
print('np.log10(Am) raises warning')
t = np.log10(Am)
print(t.data)
print(t.mask)

# Now my similar issue
np.seterr(all='raise') # provides full traceback
x = np.ma.array([1.0, np.nan, 3.0], mask=[False, True, False])
print(x)
print(1/x) # generates an error
```

## Output

```
2.7.3 (default, Mar 13 2014, 11:03:55)
[GCC 4.7.2]
np.__version__ 1.9.2
np.ma.log10(Am) works OK.
[-1.      -1.       0.       0.       0.       0.       0.30103]
[ True  True  True  True False False False]
np.log10(Am) raises warning
ma_error.py:15: RuntimeWarning: divide by zero encountered in log10
  t = np.log10(Am)
ma_error.py:15: RuntimeWarning: invalid value encountered in log10
  t = np.log10(Am)
[ 1.       1.       1.       1.       0.       0.       0.30103]
[ True  True  True  True False False False]
[1.0 -- 3.0]
Traceback (most recent call last):
  File "ma_error.py", line 23, in <module>
    print(1/x) # generates an error
  File "~/.local/lib/python2.7/site-packages/numpy/ma/core.py", line 2875, in __array_wrap__
    d = filled(domain(*args), True)
  File "~/.local/lib/python2.7/site-packages/numpy/ma/core.py", line 790, in __call__
    return umath.absolute(a) * self.tolerance >= umath.absolute(b)
FloatingPointError: invalid value encountered in greater_equal
```

## Notes

I had a quick look at passing the mask (OR of the input values masks) to the domain function and only testing non-masked values. Something like for line 790 

``` python
ret = np.oneslike(m, dtype=np.bool_)
ret[~m] = umath.absolute(a[~m]) * self.tolerance >= umath.absolute(b[~m])
return ret
```

Just to summarize, there are two issues.  One is that `np.log(m)` has different behavior than `np.ma.log(m)` when `m` is a masked array.  A separate issue is that `np.ma.log` has trouble with masked-out `nan` although it can deal intelligently with other masked-out values outside of the domain of `log`.  These issues generalize to functions other than `log`.

``` python
>>> import numpy as np
>>> np.log(np.ma.array([1, -1], mask=[0, 1]))
__main__:1: RuntimeWarning: invalid value encountered in log
masked_array(data = [0.0 --],
             mask = [False  True],
       fill_value = 1e+20)

>>> np.ma.log(np.ma.array([1, np.nan], mask=[0, 1]))
/numpy/ma/core.py:805: RuntimeWarning: invalid value encountered in less_equal
  return umath.less_equal(x, self.critical_value)
masked_array(data = [0.0 --],
             mask = [False  True],
       fill_value = 1e+20)
```

On the second issue, it is not just np.ma.log10 but a number of other functions that don't handle np.nan values in masked elements.

This issue still exist for Numpy version 1.11.2
The problem also appears in simple comparisons:
```
In [2]: np.ma.masked_invalid([1.1, np.nan]) > 1
/Users/efiring/anaconda/envs/test/bin/ipython:1: RuntimeWarning: invalid value encountered in greater
  #!/Users/efiring/anaconda/envs/test/bin/python

masked_array(data = [True --],
             mask = [False  True],
       fill_value = True)
```
How would I do the following:
```
d = np.ma.array([1, None], mask=[False, True])
d[d > 0] =  0
```

This currently, creates a  `TypeError: '>' not supported between instances of 'NoneType' and 'int'`.

I love numpy and think masked arrays are a great thing to have, but not having this fixed makes them quite useless in many cases.

More than happy to help, if anybody provides me with some pointers.

**Edit**
Seems to be an issue here: https://github.com/numpy/numpy/blob/efa7bad30d92f5f58448e753b3d280d3bea157be/numpy/ma/core.py#L4045-L4061
Looks like a fix has been implemented for structured arrays, maybe it is enough to default to the `if` statement?

**Edit 2**
I was wrong, this is only called for `__eq__` and `__ne__`. The `__lt__`, `__le__`, `__gt__` and `__ge__` methods seem to be not defined for a masked array.

**Edit 3**
Found a bit hacky solution for the issue, if someone has the same:

```
tmp = d.data[~d.mask]
tmp[tmp > 0] = 0
d.data[~d.mask] = tmp
```
@cosama: please open a new issue, that seems unrelated
@eric-wieser. I can do that, but if I change `None` to `np.nan` in the example I provided, it causes the warning issue mentioned here https://github.com/numpy/numpy/issues/4959#issuecomment-272966011. It is just a more extreme case, of the issue, where instead of causing a warning it actually breaks.

**Edit:**

I opened a new issue concerning the comparision operations here: https://github.com/numpy/numpy/issues/15978. So that the discussion here can focus on the `ufunc` implementations.
I've been running into this warning when passing values to a log-scaled axis in Matplotlib, where I've masked the bad values before passing in, so I expect everything to be OK.

The problem is that `np.log10(x)` will call the standard ufunc even if x is a `MaskedArray`. The warning is avoided if instead, I call `np.ma.log10(x)` which then goes into the Masked ufunc implementation.

@efiring, I started using your simple example, but it is actually fixed in master as a side-effect of #15230. 

I think the action item would be that if a ufunc gets a MaskedArray as input, that it dispatches over to the Masked Implementation of that ufunc if it exists. I'm not sure if that was an explicit design decision or not though.
It looks like `MaskedArray` does not define a `__array_ufunc__` function, which could be used to do the dispatching to the local masked functions in the module. I just tried this quick prototype, which seemed to solve this issue.
```python
    def __array_ufunc__(self, ufunc, method, *inputs, **kwargs):
        HANDLED_FUNCTIONS = {'log10': log10}
        func = HANDLED_FUNCTIONS.get(ufunc.__name__, None)
        if func:
            return getattr(func, method)(*inputs, **kwargs)
        return NotImplemented
```

Is this something that is worth implementing on the `MaskedArray` class? If so, I'd be happy to try and write up a PR to generalize this to all ufuncs in the `np.ma` module already.

I tried to see if anyone had worked on this before, but it doesn't look like it. This idea was brought up in #15200 suggesting this may be a good thing to do.
ping @mhvk who may have some thoughts here.

I _suspect_ the right fix is to define a `__array_ufunc__` method on `MaskedArray`, and rewrite the arithmetic special methods from:
```python
   def add(self, other):
        if self._delegate_binop(other):
            return NotImplemented
        return add(self, other)
```
to
```python
   def add(self, other):
        if self._delegate_binop(other):
            return NotImplemented
        return np.add(self, other)
```

Currently MaskedArray overrides ufuncs via `__array_finalize__`, which is quite indirect. There is an `__array_ufunc__` method defined, but only via inheritance from the base `np.ndarray` class.
Yes, `MaskedArray` should gain a proper `__array_ufunc__`...  Right now, you also have to use the "old" system of overrides, and write a custom `__array_wrap__`. I think this may be do-able, but obviously less clear if it is worth the effort.

Resolving patch:
diff --git a/numpy/ma/core.py b/numpy/ma/core.py
--- a/numpy/ma/core.py
+++ b/numpy/ma/core.py
@@ -22,6 +22,7 @@
 # pylint: disable-msg=E1002
 import builtins
 import inspect
+from numbers import Number
 import operator
 import warnings
 import textwrap
@@ -58,10 +59,10 @@
     'frombuffer', 'fromflex', 'fromfunction', 'getdata', 'getmask',
     'getmaskarray', 'greater', 'greater_equal', 'harden_mask', 'hypot',
     'identity', 'ids', 'indices', 'inner', 'innerproduct', 'isMA',
-    'isMaskedArray', 'is_mask', 'is_masked', 'isarray', 'left_shift',
-    'less', 'less_equal', 'log', 'log10', 'log2',
-    'logical_and', 'logical_not', 'logical_or', 'logical_xor', 'make_mask',
-    'make_mask_descr', 'make_mask_none', 'mask_or', 'masked',
+    'isMaskedArray', 'is_mask', 'is_masked', 'isarray', 'isfinite',
+    'isinf', 'isnan', 'left_shift', 'less', 'less_equal', 'log', 'log10',
+    'log2', 'logical_and', 'logical_not', 'logical_or', 'logical_xor',
+    'make_mask', 'make_mask_descr', 'make_mask_none', 'mask_or', 'masked',
     'masked_array', 'masked_equal', 'masked_greater',
     'masked_greater_equal', 'masked_inside', 'masked_invalid',
     'masked_less', 'masked_less_equal', 'masked_not_equal',
@@ -925,6 +926,12 @@ def __call__(self, a, *args, **kwargs):
 
         """
         d = getdata(a)
+        if 'out' in kwargs:
+            # Need to drop the mask from the output array when being called
+            kwargs['out'] = getdata(kwargs['out'])
+        args = [getdata(arg) if isinstance(arg, MaskedArray) else arg
+                for arg in args]
+
         # Deal with domain
         if self.domain is not None:
             # Case 1.1. : Domained function
@@ -1048,7 +1055,7 @@ def __call__(self, a, b, *args, **kwargs):
             masked_result._update_from(b)
         return masked_result
 
-    def reduce(self, target, axis=0, dtype=None):
+    def reduce(self, target, axis=0, dtype=None, **kwargs):
         """
         Reduce `target` along the given `axis`.
 
@@ -1183,6 +1190,10 @@ def __call__(self, a, b, *args, **kwargs):
 
         # Transforms to a (subclass of) MaskedArray
         masked_result = result.view(get_masked_subclass(a, b))
+        # If the original masks were scalar or nomask, don't expand the result
+        # which comes from the isfinite initialization above
+        if getmask(a).shape + getmask(b).shape == ():
+            m = _shrink_mask(m)
         masked_result._mask = m
         if isinstance(a, MaskedArray):
             masked_result._update_from(a)
@@ -1209,6 +1220,9 @@ def __call__(self, a, b, *args, **kwargs):
 ceil = _MaskedUnaryOperation(umath.ceil)
 around = _MaskedUnaryOperation(np.round_)
 logical_not = _MaskedUnaryOperation(umath.logical_not)
+isinf = _MaskedUnaryOperation(umath.isinf)
+isnan = _MaskedUnaryOperation(umath.isnan)
+isfinite = _MaskedUnaryOperation(umath.isfinite)
 
 # Domained unary ufuncs
 sqrt = _MaskedUnaryOperation(umath.sqrt, 0.0,
@@ -3081,7 +3095,7 @@ def __array_wrap__(self, obj, context=None):
             func, args, out_i = context
             # args sometimes contains outputs (gh-10459), which we don't want
             input_args = args[:func.nin]
-            m = reduce(mask_or, [getmaskarray(arg) for arg in input_args])
+            m = reduce(mask_or, [getmask(arg) for arg in input_args])
             # Get the domain mask
             domain = ufunc_domain.get(func, None)
             if domain is not None:
@@ -3109,7 +3123,6 @@ def __array_wrap__(self, obj, context=None):
                     else:
                         # Don't modify inplace, we risk back-propagation
                         m = (m | d)
-
             # Make sure the mask has the proper size
             if result is not self and result.shape == () and m:
                 return masked
@@ -3119,6 +3132,85 @@ def __array_wrap__(self, obj, context=None):
 
         return result
 
+    def __array_ufunc__(self, np_ufunc, method, *inputs, **kwargs):
+        """
+        MaskedArray capability for ufuncs
+
+        Handle masked versions of ufuncs if they are implemented within
+        the MaskedArray module. If the masked ufunc is not implemented,
+        this falls back to the standard numpy ndarray ufunc, which we
+        then call with the ndarray view of the input data.
+
+        """
+        # Output can be specified as arguments or as keyword arguments
+        outputs = kwargs.pop('out', ())
+        if not isinstance(outputs, tuple):
+            outputs = (outputs,)
+        outputs += inputs[np_ufunc.nin:]
+        args = inputs[:np_ufunc.nin]
+
+        # Determine what class types we are compatible with and return
+        # NotImplemented if we don't know how to handle them
+        for arg in args + outputs:
+            if not isinstance(arg, (ndarray, np.bool_, Number, list, str)):
+                return NotImplemented
+
+        # Get the equivalent masked version of the numpy function
+        # if it is in the module level functions
+        ma_ufunc = np.ma.__dict__.get(np_ufunc.__name__, np_ufunc)
+        if ma_ufunc is np_ufunc:
+            # We didn't have a Masked version of the ufunc, so we need to
+            # call the ndarray version with the data from the objects and
+            # prevent infinite recursion.
+
+            # Make ndarray views of the input arguments
+            args = [getdata(input) if isinstance(input, MaskedArray)
+                    else input for input in args]
+        else:
+            # The masked power function doesn't support extra args
+            if np_ufunc.__name__ in ('power'):
+                kwargs = {}
+
+        results = getattr(ma_ufunc, method)(*args, **kwargs)
+        if results is NotImplemented:
+            return NotImplemented
+        if method == 'at':
+            return
+        if np_ufunc.nout == 1:
+            results = (results,)
+        if outputs == ():
+            outputs = (None,) * np_ufunc.nout
+
+        returns = []
+        for i, result in enumerate(results):
+            output = outputs[i]
+
+            # Reapply the mask
+            if isinstance(result, ndarray) and result is not masked:
+                # Need to copy over all of the data and mask from results
+                # to the original object requested with out
+                if output is not None:
+                    if isinstance(output, MaskedArray):
+                        output._update_from(result)
+                        if isinstance(result, MaskedArray):
+                            output.data[:] = result._data
+                            output._mask = result._mask
+                        else:
+                            output.data[:] = result
+                    else:
+                        output[:] = result
+
+                    result = output
+
+            elif output is not None:
+                # An out value was requested, but the result is a scalar
+                output[()] = result
+                result = output
+
+            returns.append(result)
+
+        return returns[0] if np_ufunc.nout == 1 else returns
+
     def view(self, dtype=None, type=None, fill_value=None):
         """
         Return a view of the MaskedArray data.
@@ -3292,7 +3384,7 @@ def _scalar_heuristic(arr, elem):
                     return dout
         else:
             # Force dout to MA
-            dout = dout.view(type(self))
+            dout = MaskedArray(dout)
             # Inherit attributes from self
             dout._update_from(self)
             # Check the fill_value
@@ -3854,6 +3946,23 @@ def filled(self, fill_value=None):
                     result = self._data
         return result
 
+    def clip(self, a_min, a_max, out=None, **kwargs):
+        """docstring inherited
+        np.clip.__doc__
+
+        TODO: Should we ignore the clip where the data is masked?
+        It is currently in line with the old numpy version
+        """
+        result = self.data.clip(a_min, a_max, **kwargs).view(MaskedArray)
+        if out is not None:
+            # Just copy the data and mask
+            out.data[:] = getdata(result)
+            out._mask = self._mask
+            return out
+        result._update_from(self)
+        result._mask = self._mask
+        return result
+
     def compressed(self):
         """
         Return all the non-masked data as a 1-D array.
@@ -3947,10 +4056,15 @@ def compress(self, condition, axis=None, out=None):
         # values.
         condition = np.asarray(condition)
 
-        _new = _data.compress(condition, axis=axis, out=out).view(type(self))
+        _new = _data.compress(condition, axis=axis).view(type(self))
         _new._update_from(self)
         if _mask is not nomask:
             _new._mask = _mask.compress(condition, axis=axis)
+        if out is not None:
+            out._update_from(self)
+            out.data[:] = _new.data
+            out._mask = _new.mask
+            return out
         return _new
 
     def _insert_masked_print(self):
@@ -4200,7 +4314,7 @@ def __add__(self, other):
         """
         if self._delegate_binop(other):
             return NotImplemented
-        return add(self, other)
+        return np.add(self, other)
 
     def __radd__(self, other):
         """
@@ -4209,7 +4323,7 @@ def __radd__(self, other):
         """
         # In analogy with __rsub__ and __rdiv__, use original order:
         # we get here from `other + self`.
-        return add(other, self)
+        return np.add(other, self)
 
     def __sub__(self, other):
         """
@@ -4218,20 +4332,20 @@ def __sub__(self, other):
         """
         if self._delegate_binop(other):
             return NotImplemented
-        return subtract(self, other)
+        return np.subtract(self, other)
 
     def __rsub__(self, other):
         """
         Subtract self from other, and return a new masked array.
 
         """
-        return subtract(other, self)
+        return np.subtract(other, self)
 
     def __mul__(self, other):
         "Multiply self by other, and return a new masked array."
         if self._delegate_binop(other):
             return NotImplemented
-        return multiply(self, other)
+        return np.multiply(self, other)
 
     def __rmul__(self, other):
         """
@@ -4240,7 +4354,7 @@ def __rmul__(self, other):
         """
         # In analogy with __rsub__ and __rdiv__, use original order:
         # we get here from `other * self`.
-        return multiply(other, self)
+        return np.multiply(other, self)
 
     def __div__(self, other):
         """
@@ -4249,7 +4363,7 @@ def __div__(self, other):
         """
         if self._delegate_binop(other):
             return NotImplemented
-        return divide(self, other)
+        return np.divide(self, other)
 
     def __truediv__(self, other):
         """
@@ -4258,14 +4372,14 @@ def __truediv__(self, other):
         """
         if self._delegate_binop(other):
             return NotImplemented
-        return true_divide(self, other)
+        return np.true_divide(self, other)
 
     def __rtruediv__(self, other):
         """
         Divide self into other, and return a new masked array.
 
         """
-        return true_divide(other, self)
+        return np.true_divide(other, self)
 
     def __floordiv__(self, other):
         """
@@ -4274,14 +4388,14 @@ def __floordiv__(self, other):
         """
         if self._delegate_binop(other):
             return NotImplemented
-        return floor_divide(self, other)
+        return np.floor_divide(self, other)
 
     def __rfloordiv__(self, other):
         """
         Divide self into other, and return a new masked array.
 
         """
-        return floor_divide(other, self)
+        return np.floor_divide(other, self)
 
     def __pow__(self, other):
         """
@@ -5058,8 +5172,8 @@ def trace(self, offset=0, axis1=0, axis2=1, dtype=None, out=None):
         #!!!: implement out + test!
         m = self._mask
         if m is nomask:
-            result = super().trace(offset=offset, axis1=axis1, axis2=axis2,
-                                   out=out)
+            result = self.view(np.ndarray).trace(offset=offset, axis1=axis1,
+                                                    axis2=axis2, out=out)
             return result.astype(dtype)
         else:
             D = self.diagonal(offset=offset, axis1=axis1, axis2=axis2)
@@ -5159,7 +5273,9 @@ def sum(self, axis=None, dtype=None, out=None, keepdims=np._NoValue):
                 result = masked
             return result
         # Explicit output
-        result = self.filled(0).sum(axis, dtype=dtype, out=out, **kwargs)
+
+        self.filled(0).sum(axis, dtype=dtype, out=out.view(np.ndarray),
+                           **kwargs)
         if isinstance(out, MaskedArray):
             outmask = getmask(out)
             if outmask is nomask:
@@ -5309,7 +5425,10 @@ def mean(self, axis=None, dtype=None, out=None, keepdims=np._NoValue):
         """
         kwargs = {} if keepdims is np._NoValue else {'keepdims': keepdims}
         if self._mask is nomask:
-            result = super().mean(axis=axis, dtype=dtype, **kwargs)[()]
+            result = self.view(np.ndarray).mean(axis=axis,
+                                                   dtype=dtype, **kwargs)
+            if isinstance(result, np.ndarray):
+                result = MaskedArray(result, mask=False)
         else:
             is_float16_result = False
             if dtype is None:
@@ -5392,9 +5511,12 @@ def var(self, axis=None, dtype=None, out=None, ddof=0,
 
         # Easy case: nomask, business as usual
         if self._mask is nomask:
-            ret = super().var(axis=axis, dtype=dtype, out=out, ddof=ddof,
-                              **kwargs)[()]
+            ret = self.view(np.ndarray).var(axis=axis, dtype=dtype,
+                                            ddof=ddof, **kwargs)
+            if isinstance(ret, np.ndarray):
+                ret = MaskedArray(ret, mask=False)
             if out is not None:
+                out.flat = ret
                 if isinstance(out, MaskedArray):
                     out.__setmask__(nomask)
                 return out
@@ -5452,12 +5574,10 @@ def std(self, axis=None, dtype=None, out=None, ddof=0,
         numpy.std : Equivalent function
         """
         kwargs = {} if keepdims is np._NoValue else {'keepdims': keepdims}
-
         dvar = self.var(axis, dtype, out, ddof, **kwargs)
         if dvar is not masked:
             if out is not None:
-                np.power(out, 0.5, out=out, casting='unsafe')
-                return out
+                return np.power(out, 0.5, out=out, casting='unsafe')
             dvar = sqrt(dvar)
         return dvar
 
@@ -5472,6 +5592,10 @@ def round(self, decimals=0, out=None):
         numpy.ndarray.round : corresponding function for ndarrays
         numpy.around : equivalent function
         """
+        stored_out = None
+        if isinstance(out, MaskedArray):
+            stored_out = out
+            out = getdata(out)
         result = self._data.round(decimals=decimals, out=out).view(type(self))
         if result.ndim > 0:
             result._mask = self._mask
@@ -5482,7 +5606,9 @@ def round(self, decimals=0, out=None):
         # No explicit output: we're done
         if out is None:
             return result
-        if isinstance(out, MaskedArray):
+        if stored_out is not None:
+            # We got in a masked array originally, so we need to return one
+            out = stored_out
             out.__setmask__(self._mask)
         return out
 
@@ -5870,7 +5996,7 @@ def mini(self, axis=None):
             "`mini` is deprecated; use the `min` method or "
             "`np.ma.minimum.reduce instead.",
             DeprecationWarning, stacklevel=2)
-        return minimum.reduce(self, axis)
+        return MaskedArray(np.min(self, axis))
 
     def max(self, axis=None, out=None, fill_value=None, keepdims=np._NoValue):
         """
@@ -6034,13 +6160,13 @@ def partition(self, *args, **kwargs):
         warnings.warn("Warning: 'partition' will ignore the 'mask' "
                       f"of the {self.__class__.__name__}.",
                       stacklevel=2)
-        return super().partition(*args, **kwargs)
+        return self.view(np.ndarray).partition(*args, **kwargs)
 
     def argpartition(self, *args, **kwargs):
         warnings.warn("Warning: 'argpartition' will ignore the 'mask' "
                       f"of the {self.__class__.__name__}.",
                       stacklevel=2)
-        return super().argpartition(*args, **kwargs)
+        return self.view(np.ndarray).argpartition(*args, **kwargs)
 
     def take(self, indices, axis=None, out=None, mode='raise'):
         """
@@ -6730,7 +6856,7 @@ def __call__(self, a, b=None):
             return self.reduce(a)
         return where(self.compare(a, b), a, b)
 
-    def reduce(self, target, axis=np._NoValue):
+    def reduce(self, target, axis=np._NoValue, **kwargs):
         "Reduce target along the given axis."
         target = narray(target, copy=False, subok=True)
         m = getmask(target)
@@ -6745,12 +6871,10 @@ def reduce(self, target, axis=np._NoValue):
             axis = None
 
         if axis is not np._NoValue:
-            kwargs = dict(axis=axis)
-        else:
-            kwargs = dict()
+            kwargs['axis'] = axis
 
         if m is nomask:
-            t = self.f.reduce(target, **kwargs)
+            t = self.f.reduce(target.view(np.ndarray), **kwargs)
         else:
             target = target.filled(
                 self.fill_value_func(target)).view(type(target))
@@ -7954,6 +8078,23 @@ def allclose(a, b, masked_equal=True, rtol=1e-5, atol=1e-8):
     """
     x = masked_array(a, copy=False)
     y = masked_array(b, copy=False)
+    if masked_equal:
+        # Apply the combined mask right away to avoid comparisons at the
+        # masked locations (assumed mask is True)
+        m = mask_or(getmask(x), getmask(y))
+        # Expand scalars to the proper dimension for comparison if needed
+        if shape(x) != shape(y):
+            if size(x) == 1:
+                # scalar a
+                x = masked_array(np.ones(shape=shape(y))*x, mask=m)
+            elif size(y) == 1:
+                # scalar b
+                y = masked_array(np.ones(shape=shape(x))*y, mask=m)
+            else:
+                raise ValueError("Cannot compare arrays of different shapes.")
+        else:
+            x = masked_array(a, copy=False, mask=m)
+            y = masked_array(b, copy=False, mask=m)
 
     # make sure y is an inexact type to avoid abs(MIN_INT); will cause
     # casting of x later.
@@ -7966,8 +8107,7 @@ def allclose(a, b, masked_equal=True, rtol=1e-5, atol=1e-8):
         if y.dtype != dtype:
             y = masked_array(y, dtype=dtype, copy=False)
 
-    m = mask_or(getmask(x), getmask(y))
-    xinf = np.isinf(masked_array(x, copy=False, mask=m)).filled(False)
+    xinf = filled(np.isinf(x), False)
     # If we have some infs, they should fall at the same place.
     if not np.all(xinf == filled(np.isinf(y), False)):
         return False
diff --git a/numpy/ma/extras.py b/numpy/ma/extras.py
--- a/numpy/ma/extras.py
+++ b/numpy/ma/extras.py
@@ -588,8 +588,8 @@ def average(a, axis=None, weights=None, returned=False, *,
     >>> avg, sumweights = np.ma.average(x, axis=0, weights=[1, 2, 3],
     ...                                 returned=True)
     >>> avg
-    masked_array(data=[2.6666666666666665, 3.6666666666666665],
-                 mask=[False, False],
+    masked_array(data=[2.66666667, 3.66666667],
+                 mask=False,
            fill_value=1e+20)
 
     With ``keepdims=True``, the following result has shape (3, 1).
@@ -2016,8 +2016,15 @@ def polyfit(x, y, deg, rcond=None, full=False, w=None, cov=False):
         not_m = ~m
         if w is not None:
             w = w[not_m]
-        return np.polyfit(x[not_m], y[not_m], deg, rcond, full, w, cov)
-    else:
-        return np.polyfit(x, y, deg, rcond, full, w, cov)
+        x = x[not_m]
+        y = y[not_m]
+
+    # Only pass the ndarray data
+    if w is not None:
+        w = w.view(np.ndarray)
+    x = x.view(np.ndarray)
+    y = y.view(np.ndarray)
+
+    return np.polyfit(x, y, deg, rcond, full, w, cov)
 
 polyfit.__doc__ = ma.doc_note(np.polyfit.__doc__, polyfit.__doc__)


Test patch:
diff --git a/numpy/lib/tests/test_function_base.py b/numpy/lib/tests/test_function_base.py
--- a/numpy/lib/tests/test_function_base.py
+++ b/numpy/lib/tests/test_function_base.py
@@ -780,7 +780,7 @@ def test_subclass(self):
                      mask=[[False, False], [True, False],
                            [False, True], [True, True], [False, False]])
         out = diff(x)
-        assert_array_equal(out.data, [[1], [1], [1], [1], [1]])
+        assert_array_equal(out.data, [[1], [4], [6], [8], [1]])
         assert_array_equal(out.mask, [[False], [True],
                                       [True], [True], [False]])
         assert_(type(out) is type(x))
diff --git a/numpy/ma/tests/test_core.py b/numpy/ma/tests/test_core.py
--- a/numpy/ma/tests/test_core.py
+++ b/numpy/ma/tests/test_core.py
@@ -3208,7 +3208,7 @@ def test_compress(self):
         assert_equal(b.fill_value, 9999)
         assert_equal(b, a[condition])
 
-        condition = (a < 4.)
+        condition = (a.data < 4.)
         b = a.compress(condition)
         assert_equal(b._data, [1., 2., 3.])
         assert_equal(b._mask, [0, 0, 1])
@@ -5367,7 +5367,7 @@ def test_ufunc_with_out_varied():
     a        = array([ 1,  2,  3], mask=[1, 0, 0])
     b        = array([10, 20, 30], mask=[1, 0, 0])
     out      = array([ 0,  0,  0], mask=[0, 0, 1])
-    expected = array([11, 22, 33], mask=[1, 0, 0])
+    expected = array([1, 22, 33], mask=[1, 0, 0])
 
     out_pos = out.copy()
     res_pos = np.add(a, b, out_pos)
diff --git a/numpy/ma/tests/test_subclassing.py b/numpy/ma/tests/test_subclassing.py
--- a/numpy/ma/tests/test_subclassing.py
+++ b/numpy/ma/tests/test_subclassing.py
@@ -7,6 +7,7 @@
 
 """
 import numpy as np
+from numpy.lib.mixins import NDArrayOperatorsMixin
 from numpy.testing import assert_, assert_raises
 from numpy.ma.testutils import assert_equal
 from numpy.ma.core import (
@@ -147,6 +148,33 @@ def __array_wrap__(self, obj, context=None):
         return obj
 
 
+class WrappedArray(NDArrayOperatorsMixin):
+    """
+    Wrapping a MaskedArray rather than subclassing to test that
+    ufunc deferrals are commutative.
+    See: https://github.com/numpy/numpy/issues/15200)
+    """
+    __array_priority__ = 20
+
+    def __init__(self, array, **attrs):
+        self._array = array
+        self.attrs = attrs
+
+    def __repr__(self):
+        return f"{self.__class__.__name__}(\n{self._array}\n{self.attrs}\n)"
+
+    def __array__(self):
+        return np.asarray(self._array)
+
+    def __array_ufunc__(self, ufunc, method, *inputs, **kwargs):
+        if method == '__call__':
+            inputs = [arg._array if isinstance(arg, self.__class__) else arg
+                      for arg in inputs]
+            return self.__class__(ufunc(*inputs, **kwargs), **self.attrs)
+        else:
+            return NotImplemented
+
+
 class TestSubclassing:
     # Test suite for masked subclasses of ndarray.
 
@@ -384,3 +412,34 @@ def test_array_no_inheritance():
     # Test that the mask is False and not shared when keep_mask=False
     assert_(not new_array.mask)
     assert_(not new_array.sharedmask)
+
+
+class TestClassWrapping:
+    # Test suite for classes that wrap MaskedArrays
+
+    def setup(self):
+        m = np.ma.masked_array([1, 3, 5], mask=[False, True, False])
+        wm = WrappedArray(m)
+        self.data = (m, wm)
+
+    def test_masked_unary_operations(self):
+        # Tests masked_unary_operation
+        (m, wm) = self.data
+        with np.errstate(divide='ignore'):
+            assert_(isinstance(np.log(wm), WrappedArray))
+
+    def test_masked_binary_operations(self):
+        # Tests masked_binary_operation
+        (m, wm) = self.data
+        # Result should be a WrappedArray
+        assert_(isinstance(np.add(wm, wm), WrappedArray))
+        assert_(isinstance(np.add(m, wm), WrappedArray))
+        assert_(isinstance(np.add(wm, m), WrappedArray))
+        # add and '+' should call the same ufunc
+        assert_equal(np.add(m, wm), m + wm)
+        assert_(isinstance(np.hypot(m, wm), WrappedArray))
+        assert_(isinstance(np.hypot(wm, m), WrappedArray))
+        # Test domained binary operations
+        assert_(isinstance(np.divide(wm, m), WrappedArray))
+        assert_(isinstance(np.divide(m, wm), WrappedArray))
+        assert_equal(np.divide(wm, m) * m, np.divide(m, m) * wm)