Repository: numpy/numpy

Issue:
mean() (and median()) should work with "object" arrays
With NumPy 1.8, `mean()` started to break when calculating the (global) mean of an array that contains objects (arrays with an object `dtype`). This also breaks `median()` on such arrays. Here is an example:

```
>>> numpy.arange(10).astype(object).mean()
Traceback (most recent call last):
  File "<ipython-input-11-782b7c0104c3>", line 1, in <module>
    numpy.arange(10).astype(object).mean()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/_methods.py", line 67, in _mean
    ret = ret.dtype.type(ret / rcount)
AttributeError: 'int' object has no attribute 'dtype'
```

Another example is case of numbers with uncertainties from the uncertainties package (https://github.com/lebigot/uncertainties/issues/22). 

I think that it would be better if NumPy did not assume that scalar results have a `dtype`, since arrays can contain objects that have a meaningful mean. I believe that such objects should not be forced to have a `dtype`, which is obviously NumPy specific (they even can't, for Python scalars like floats). Furthermore, a `dtype` is in principle not necessary for the calculation of the mean of such objects, so it would look strange if they had to have one.

The problem is that `numpy.mean()` assumes that the intermediate result obtained has a `dtype` (with a `type` attribute).

Therefore, I suggest that NumPy's `mean()` also handle arrays of objects that are not of the standard NumPy types (their `dtype` is object, and they contain objects that have a meaningful mean, like ints, floats, numbers with uncertainties, etc.).



Discussion:
introduced in f16b12e87 by @charris

Hmmm, this is annoying. It seems to me like the only way to fix this is probably to see if `dtype` is given, and then use `np.dtype(dtype).type(...)` and otherwise just do the plain operation?

Yeah, I was thinking along the same lines.

I would be curious to see what the issue that prompted the change was, to see if I can come up with any kind of better suggestion (who knows): what was the problem, exactly?

The scalar returns didn't preserve type, i.e., float32 would go to float64. That was on account of type precedence between scalars being different than type precedence between scalars and arrays.

@charris do we care even about that? Or is it enough if the passed in dtype actually gets honored?

I honestly have troubles to figure out a good method of preserving the type quite right for the scalar result. I now think we may have to just check for object dtype input (or passed in dtype). The most secure method I can think of would be a new keyword argument to the ufuncs to skip PyArray_Return (would probably be slower though), but unless that is useful elsewhere it is not worth the trouble either.

Any progress on this one? It costed me a hour of debugging today. If doing this properly is hard, please consider fixing the error message so it is obvious what's wrong. 

@charris do you have time to have a look at this?

I also think we accumulated enough fixes to warrant a 1.8.1 release if we add this and the C99 windows fix.
thoughts?

I'll get it done today sometime. Agree on 1.8.1, I came to that conclusion this morning. We should also fix the `divide` and `true_divide` ufuncs when the `dtype` is given.

I need to think about this a bit more before putting up a fix.


Resolving patch:
diff --git a/numpy/core/_methods.py b/numpy/core/_methods.py
--- a/numpy/core/_methods.py
+++ b/numpy/core/_methods.py
@@ -63,8 +63,10 @@ def _mean(a, axis=None, dtype=None, out=None, keepdims=False):
     if isinstance(ret, mu.ndarray):
         ret = um.true_divide(
                 ret, rcount, out=ret, casting='unsafe', subok=False)
-    else:
+    elif hasattr(ret, 'dtype'):
         ret = ret.dtype.type(ret / rcount)
+    else:
+        ret = ret / rcount
 
     return ret
 
@@ -107,8 +109,10 @@ def _var(a, axis=None, dtype=None, out=None, ddof=0, keepdims=False):
     if isinstance(ret, mu.ndarray):
         ret = um.true_divide(
                 ret, rcount, out=ret, casting='unsafe', subok=False)
-    else:
+    elif hasattr(ret, 'dtype'):
         ret = ret.dtype.type(ret / rcount)
+    else:
+        ret = ret / rcount
 
     return ret
 
@@ -118,7 +122,9 @@ def _std(a, axis=None, dtype=None, out=None, ddof=0, keepdims=False):
 
     if isinstance(ret, mu.ndarray):
         ret = um.sqrt(ret, out=ret)
-    else:
+    elif hasattr(ret, 'dtype'):
         ret = ret.dtype.type(um.sqrt(ret))
+    else:
+        ret = um.sqrt(ret)
 
     return ret


Test patch:
diff --git a/numpy/core/tests/test_multiarray.py b/numpy/core/tests/test_multiarray.py
--- a/numpy/core/tests/test_multiarray.py
+++ b/numpy/core/tests/test_multiarray.py
@@ -10,6 +10,8 @@
     import builtins
 else:
     import __builtin__ as builtins
+from decimal import Decimal
+
 
 import numpy as np
 from nose import SkipTest
@@ -2840,6 +2842,8 @@ def setUp(self):
         np.random.seed(range(3))
         self.rmat = np.random.random((4, 5))
         self.cmat = self.rmat + 1j * self.rmat
+        self.omat = np.array([Decimal(repr(r)) for r in self.rmat.flat])
+        self.omat = self.omat.reshape(4, 5)
 
     def test_keepdims(self):
         mat = np.eye(3)
@@ -2866,9 +2870,20 @@ def test_out(self):
         assert_raises(ValueError, f, mat, axis=1, out=out)
 
     def test_dtype_from_input(self):
+
         icodes = np.typecodes['AllInteger']
         fcodes = np.typecodes['AllFloat']
 
+        # object type
+        for f in self.funcs:
+            mat = np.array([[Decimal(1)]*3]*3)
+            tgt = mat.dtype.type
+            res = f(mat, axis=1).dtype.type
+            assert_(res is tgt)
+            # scalar case
+            res = type(f(mat, axis=None))
+            assert_(res is Decimal)
+
         # integer types
         for f in self.funcs:
             for c in icodes:
@@ -2879,6 +2894,7 @@ def test_dtype_from_input(self):
                 # scalar case
                 res = f(mat, axis=None).dtype.type
                 assert_(res is tgt)
+
         # mean for float types
         for f in [_mean]:
             for c in fcodes:
@@ -2889,10 +2905,12 @@ def test_dtype_from_input(self):
                 # scalar case
                 res = f(mat, axis=None).dtype.type
                 assert_(res is tgt)
+
         # var, std for float types
         for f in [_var, _std]:
             for c in fcodes:
                 mat = np.eye(3, dtype=c)
+                # deal with complex types
                 tgt = mat.real.dtype.type
                 res = f(mat, axis=1).dtype.type
                 assert_(res is tgt)
@@ -2968,7 +2986,7 @@ def test_empty(self):
                     assert_equal(f(A, axis=axis), np.zeros([]))
 
     def test_mean_values(self):
-        for mat in [self.rmat, self.cmat]:
+        for mat in [self.rmat, self.cmat, self.omat]:
             for axis in [0, 1]:
                 tgt = mat.sum(axis=axis)
                 res = _mean(mat, axis=axis) * mat.shape[axis]
@@ -2979,16 +2997,16 @@ def test_mean_values(self):
                 assert_almost_equal(res, tgt)
 
     def test_var_values(self):
-        for mat in [self.rmat, self.cmat]:
+        for mat in [self.rmat, self.cmat, self.omat]:
             for axis in [0, 1, None]:
                 msqr = _mean(mat * mat.conj(), axis=axis)
                 mean = _mean(mat, axis=axis)
-                tgt = msqr - mean * mean.conj()
+                tgt = msqr - mean * mean.conjugate()
                 res = _var(mat, axis=axis)
                 assert_almost_equal(res, tgt)
 
     def test_std_values(self):
-        for mat in [self.rmat, self.cmat]:
+        for mat in [self.rmat, self.cmat, self.omat]:
             for axis in [0, 1, None]:
                 tgt = np.sqrt(_var(mat, axis=axis))
                 res = _std(mat, axis=axis)
@@ -3116,12 +3134,6 @@ def test_broadcast2(self):
         A = np.choose(self.ind, (self.x, self.y2))
         assert_equal(A, [[2, 2, 3], [2, 2, 3]])
 
-def can_use_decimal():
-    try:
-        from decimal import Decimal
-        return True
-    except ImportError:
-        return False
 
 # TODO: test for multidimensional
 NEIGH_MODE = {'zero': 0, 'one': 1, 'constant': 2, 'circular': 3, 'mirror': 4}
@@ -3157,11 +3169,7 @@ def _test_simple2d(self, dt):
     def test_simple2d(self):
         self._test_simple2d(np.float)
 
-    @dec.skipif(not can_use_decimal(),
-            "Skip neighborhood iterator tests for decimal objects " \
-            "(decimal module not available")
     def test_simple2d_object(self):
-        from decimal import Decimal
         self._test_simple2d(Decimal)
 
     def _test_mirror2d(self, dt):
@@ -3177,11 +3185,7 @@ def _test_mirror2d(self, dt):
     def test_mirror2d(self):
         self._test_mirror2d(np.float)
 
-    @dec.skipif(not can_use_decimal(),
-            "Skip neighborhood iterator tests for decimal objects " \
-            "(decimal module not available")
     def test_mirror2d_object(self):
-        from decimal import Decimal
         self._test_mirror2d(Decimal)
 
     # Simple, 1d tests
@@ -3203,11 +3207,7 @@ def _test_simple(self, dt):
     def test_simple_float(self):
         self._test_simple(np.float)
 
-    @dec.skipif(not can_use_decimal(),
-            "Skip neighborhood iterator tests for decimal objects " \
-            "(decimal module not available")
     def test_simple_object(self):
-        from decimal import Decimal
         self._test_simple(Decimal)
 
     # Test mirror modes
@@ -3222,11 +3222,7 @@ def _test_mirror(self, dt):
     def test_mirror(self):
         self._test_mirror(np.float)
 
-    @dec.skipif(not can_use_decimal(),
-            "Skip neighborhood iterator tests for decimal objects " \
-            "(decimal module not available")
     def test_mirror_object(self):
-        from decimal import Decimal
         self._test_mirror(Decimal)
 
     # Circular mode
@@ -3240,11 +3236,7 @@ def _test_circular(self, dt):
     def test_circular(self):
         self._test_circular(np.float)
 
-    @dec.skipif(not can_use_decimal(),
-            "Skip neighborhood iterator tests for decimal objects " \
-            "(decimal module not available")
     def test_circular_object(self):
-        from decimal import Decimal
         self._test_circular(Decimal)
 
 # Test stacking neighborhood iterators