Repository: numpy/numpy

Issue:
Inconsistent behavior for ufuncs in Numpy v1.10.X
I am copying this from a discussion on the mailing list.  

I am attempting to subclass numpy.ma.MaskedArray.  I am currently using Python v2.7.10.  The problem discussed below does not occur in Numpy v1.9.2, but does occur in all versions of Numpy v1.10.x including v1.10.4.

Using mathematical operators on my subclass behaves differently than using the analogous ufunc. When using the ufunc directly (e.g. np.subtract(arr1,
 arr2)), **array_prepare**, **array_finalize**, and **array_wrap** are all called as expected, however, when using the symbolic operator (e.g. arr1-arr2) only **array_finalize** is called. As a consequence, I lose any information stored in arr._optinfo when a mathematical operator is used.
Here is a code snippet that illustrates the issue.

```
#!/bin/env python

import numpy as np
from numpy.ma import MaskedArray, nomask

class InfoArray(MaskedArray):
    def __new__(cls, info=None, data=None, mask=nomask, dtype=None, 
                copy=False, subok=True, ndmin=0, fill_value=None,
                keep_mask=True, hard_mask=None, shrink=True, **kwargs):
        obj = super(InfoArray, cls).__new__(cls, data=data, mask=mask,
                      dtype=dtype, copy=copy, subok=subok, ndmin=ndmin, 
                      fill_value=fill_value, hard_mask=hard_mask,
                      shrink=shrink, **kwargs)
        obj._optinfo['info'] = info
        return obj

    def __array_prepare__(self, out, context=None):
        print '__array_prepare__'
        return super(InfoArray, self).__array_prepare__(out, context)

    def __array_wrap__(self, out, context=None):
        print '__array_wrap__'
        return super(InfoArray, self).__array_wrap__(out, context)

    def __array_finalize__(self, obj):
        print '__array_finalize__'
        return super(InfoArray, self).__array_finalize__(obj)

if __name__ == "__main__":
    arr1 = InfoArray('test', data=[1,2,3,4,5,6])
    arr2 = InfoArray(data=[0,1,2,3,4,5])

    diff1 = np.subtract(arr1, arr2)
    print diff1._optinfo

    diff2 = arr1-arr2
    print diff2._optinfo
```

If run, the output looks like this:

```
$ python test_ma_sub.py 
#Call to np.subtract(arr1, arr2) here
__array_finalize__
__array_finalize__
__array_prepare__
__array_finalize__
__array_wrap__
__array_finalize__
{'info': 'test'}
#Executing arr1-arr2 here
__array_finalize__
{}
```

Currently I have simply downgraded to 1.9.2 to solve the problem for myself, but have been having difficulty figuring out where the difference lies between 1.9.2 and 1.10.0.



Discussion:
Looks like 3c6b6bab is the problem commit.

@mhvk Thoughts?

@charris - I'll need to look in more detail, which will not be before next week (graduate admissions...). I suspect #5864 more, though, since that directly touched the binary operations (CC @njsmith)

@mhvk I did a bisection and 3c6b6ba is guilty ;) #5864 was my first thought also.

This is getting a little deep for me at this point, though I will try to help if I can.

While reverting to v1.9.3 solves my current issues, it seems to me that the behavior is incorrect even in the 1.9.3 release.  Calling `np.subtract(arr1, arr2)` correctly enters both __array_prepare__ and __array_wrap__.  Using the subtraction operator (`arr1-arr2`) skips this functionality, which goes counter to the numpy documentation on subclassing ndarray which states that __array_prepare__ is called "before every ufunc" and __array_wrap__ is called "at the end of every ufunc".  This is a problem for code that attempts to properly use __array_prepare__ and/or __array_wrap__ to maintain metadata.  

One issue compounded the other leaving me very confused I think.

@jsolbrig Binary ops are handled a differently from ufuncs as they interact with Python, i.e., Python makes the call to the subtract method. The proper way to handle that is a current topic of ~~discussion~~ contention.

@charris Wouldn't that mean that the documentation is incorrect, then?  The subtraction operator still calls a ufunc (albeit an overridden one), but __array_prepare__ and __array_wrap__ are not called [as specified in the documentation.](http://docs.scipy.org/doc/numpy-1.10.0/reference/arrays.classes.html#numpy.class.__array_prepare__)  I would expect the behavior of the operator and the function to be exactly the same from the standpoint of subclassing from MaskedArray.

The thing is that ufunc wrapping works on the `ndarray` level, but masked arrays do _not_ get passed into the ufuncs, instead they have their own set of functions. MaskedArrays are not part of the core numpy API and as such they neither have to, nor ever did call those methods. Of course you are right to say that it would be nice if they would behave like a real numpy array, but they are unfortunatly just a tag on module, with all the problems that come with it.

EDIT: Of course you can pass them into the ufuncs, but their operators don't, they have their own API in this regard (and it basically means that subclasses trying to do a lot of things need to do even more magic probably). But I think subclasses were somewhat anticipated, so I am not quite sure.

Okay, yeah, that makes sense.  It does make me wonder, though, if __array_prepare__ and __array_wrap__ should be called inside the __call__ method of _MaskedBinaryOperation and other similar classes in order to keep consistent behavior for subclassing and make it easier for others in the future.

Sure, maybe? It's basically impossible to fix ndarray subclassing to work well, and then MaskedArray has a giant pile of hacks to make it kinda-sorta-work as an ndarray subclass, and then you're trying to subclass _that_. So... if there's an additional hack that will make your particular use case work, then might as well try it; it's unlikely to make things worse :-). (And the accidental regression versus 1.9 is certainly not OK; sorry about that.) As a piece of general advice though I'd strongly recommend trying to find some way to solve your problem that doesn't involve subclassing any of these things :-)

Yep, I've basically come to the same conclusion that subclassing MaskedArray is probably the wrong solution.  I'm going to have to think about how best to go about it, though.  Using MaskedArray the way I have been has been so convenient other than this problem.  I now have a workaround for this problem and everything seems to work the way I would hope, but it's a little worrisome that it was so difficult to achieve.

After looking at it a bit more, though, it does appear that if called in the appropriate location in _MaskedBinaryOperation, __array_prepare__ and **array_wrap** would actually work the way they are supposed to.  Do you think it would be worth me looking into how to implement the numpy-consistent behavior?

My personal policy is to leave np.ma maintenance to those who care enough to fix things. So really it's up to you to decide whether it's a worthwhile use of your time :-). We would probably merge a pull request that fixed the regression, at least.

@jsolbrig - just as a background: the PR that broke things for you tried to fix the case where the `MaskedArray` _contains_ an `ndarray` subclass, so we should try to keep that intact (which I think should not be a problem; and anyway, it has test cases...). Anyway, am quite happy to look at possible solutions.

Just so it doesn't get lost, in the thread on numpy-dev, @seberg wrote:

> Well, there was definitely a change in that code that will cause this, i.e. the code block:
> 
> ```
>         if isinstance(b, MaskedArray):
>             if isinstance(a, MaskedArray):
>                 result._update_from(a)
>             else:
>                 result._update_from(b)
>         elif isinstance(a, MaskedArray):
>             result._update_from(a)
> ```
> 
> was changed to something like:
> 
> ```
>         masked_result._update_from(result)
> ```

@mhvk Ping...

Overwhelmed by my day job right now, in particular graduate admission, but a break is at least in sight...  Hopefully still this week.

@mhvk Ping! I'd like to get another beta out this weekend, what does your schedule look like?

The break took longer in coming than expected, but now do have some time. Hopefully this weekend or Monday.

> The break took longer in coming than expected

They always do ;) 


Resolving patch:
diff --git a/numpy/ma/core.py b/numpy/ma/core.py
--- a/numpy/ma/core.py
+++ b/numpy/ma/core.py
@@ -911,7 +911,7 @@ def __call__(self, a, *args, **kwargs):
         # Transform to
         masked_result = result.view(get_masked_subclass(a))
         masked_result._mask = m
-        masked_result._update_from(result)
+        masked_result._update_from(a)
         return masked_result
 
     def __str__(self):
@@ -999,7 +999,10 @@ def __call__(self, a, b, *args, **kwargs):
         # Transforms to a (subclass of) MaskedArray
         masked_result = result.view(get_masked_subclass(a, b))
         masked_result._mask = m
-        masked_result._update_from(result)
+        if isinstance(a, MaskedArray):
+            masked_result._update_from(a)
+        elif isinstance(b, MaskedArray):
+            masked_result._update_from(b)
         return masked_result
 
     def reduce(self, target, axis=0, dtype=None):
@@ -1030,7 +1033,6 @@ def reduce(self, target, axis=0, dtype=None):
                 return tr
         masked_tr = tr.view(tclass)
         masked_tr._mask = mr
-        masked_tr._update_from(tr)
         return masked_tr
 
     def outer(self, a, b):
@@ -1056,7 +1058,6 @@ def outer(self, a, b):
             return d
         masked_d = d.view(get_masked_subclass(a, b))
         masked_d._mask = m
-        masked_d._update_from(d)
         return masked_d
 
     def accumulate(self, target, axis=0):
@@ -1068,7 +1069,6 @@ def accumulate(self, target, axis=0):
         t = filled(target, self.filly)
         result = self.f.accumulate(t, axis)
         masked_result = result.view(tclass)
-        masked_result._update_from(result)
         return masked_result
 
     def __str__(self):
@@ -1145,7 +1145,10 @@ def __call__(self, a, b, *args, **kwargs):
         # Transforms to a (subclass of) MaskedArray
         masked_result = result.view(get_masked_subclass(a, b))
         masked_result._mask = m
-        masked_result._update_from(result)
+        if isinstance(a, MaskedArray):
+            masked_result._update_from(a)
+        elif isinstance(b, MaskedArray):
+            masked_result._update_from(b)
         return masked_result
 
     def __str__(self):


Test patch:
diff --git a/numpy/ma/tests/test_subclassing.py b/numpy/ma/tests/test_subclassing.py
--- a/numpy/ma/tests/test_subclassing.py
+++ b/numpy/ma/tests/test_subclassing.py
@@ -47,6 +47,14 @@ def __iadd__(self, other):
 subarray = SubArray
 
 
+class SubMaskedArray(MaskedArray):
+    """Pure subclass of MaskedArray, keeping some info on subclass."""
+    def __new__(cls, info=None, **kwargs):
+        obj = super(SubMaskedArray, cls).__new__(cls, **kwargs)
+        obj._optinfo['info'] = info
+        return obj
+
+
 class MSubArray(SubArray, MaskedArray):
 
     def __new__(cls, data, info={}, mask=nomask):
@@ -332,6 +340,18 @@ def test_subclass_str(self):
         mxcsub = masked_array(xcsub, mask=[True, False, True, False, False])
         self.assertTrue(str(mxcsub) == 'myprefix [-- 1 -- 3 4] mypostfix')
 
+    def test_pure_subclass_info_preservation(self):
+        # Test that ufuncs and methods conserve extra information consistently;
+        # see gh-7122.
+        arr1 = SubMaskedArray('test', data=[1,2,3,4,5,6])
+        arr2 = SubMaskedArray(data=[0,1,2,3,4,5])
+        diff1 = np.subtract(arr1, arr2)
+        self.assertTrue('info' in diff1._optinfo)
+        self.assertTrue(diff1._optinfo['info'] == 'test')
+        diff2 = arr1 - arr2
+        self.assertTrue('info' in diff2._optinfo)
+        self.assertTrue(diff2._optinfo['info'] == 'test')
+
 
 ###############################################################################
 if __name__ == '__main__':