Repository: numpy/numpy

Issue:
np.average of memmap returns memmap 0d array instead of numpy scalar
``` python
import numpy as np
mm = np.memmap('/tmp/test.data', mode='w+', shape=4)
np.average(mm)
```

Output since 5ceab8f982d80c68b3b5e90697cd20df51f4a3e8 (a couple of days ago):

```
memmap(0.0)
```

Output before that (e.g. in 1.10.4):

```
0.0
```

This causes the scikit-learn using numpy master tests to fail, see this Travis [log](https://travis-ci.org/scikit-learn/scikit-learn/jobs/114973679).

We have been bitten in the past by this in scikit-learn with `.sum` that has the same surprising behaviour (see https://github.com/scikit-learn/scikit-learn/pull/6225). Shouldn't reduce operations on a memmap array always return a numpy scalar ?

cc @ogrisel.

regression (from 1.5.x) with return of operations on memmap'ed arrays (Trac #2215)
_Original ticket http://projects.scipy.org/numpy/ticket/2215 on 2012-09-14 by @yarikoptic, assigned to unknown._

As
http://projects.scipy.org/numpy/ticket/2179
stated: "Ideally one should be returning other ndarray objects, e.g. numpy scalars in the case of sum(), but this requires a larger rewrite. At least numpy version 1.5.1rc1 did this. The only time the _mmap should be kept (as I am aware of) is when slicing. "

This issue remains not fixed thus opening a separate issue for the above.  Taking into account that np.asscalar() called on scalars 
(e.g. which would be returned by np.sum on ndarrays) crashes, it becomes impossible to write code agnostic of ndarray/memmap container types.

it is also not clear why array-wide operations on mode='c' memmap'ed arrays which cannot really get reflected in the file storage, remain memmapped and do not result in normal ndarrays, but I guess that would be a separate issue...



Discussion:
The cause is the change from `a = np.asarray(a)` to `a = np.asanyarray(a)` at the start of `np.average`, which I did to preserve subclass like `np.mean` does. Previously `np.average` only preserved the `np.matrix` subclass using a special clause, but now it preserves any subclass.

While it's definitely a change in behavior we should think carefully about, the new behavior seems more consistent with the rest of numpy. If no weights are given `np.average` just computes `np.mean`, and `np.mean` preserves the subclass (including for scalars), just like ufuncs and most other numpy methods. This is so that subclasses have a chance to modify the inputs or outputs to the computation, see some of the discussion in #5819, #667.

I suggest we should keep the old `asarray` behavior temporarily with a deprecation warning, and then (sorry @lesteve) go ahead with the change to `asanyarray` in the next release. Is that reasonable?

Possibly we might also think about updating `memmap.sum` and others so they return a scalar, or making some of the changes proposed in #667. Indeed it seems weird that memmap sum/mean return a memmap array (even for non-scalar results, eg after summing over an axis).

Pretty much all the operations are weird anyway with `numpy.memmap`, for instance the product of a memmap array by a scalar is a memmap array too: 

``` python
>>> import numpy as np
>>> a = np.memmap('a.bin', shape=10, dtype=np.uint8, mode='w+')
>>> a[:] = 1

>>> b = a * 3
>>> type(b)
<class 'numpy.core.memmap.memmap'>
>>> b.base
array([3, 3, 3, 3, 3, 3, 3, 3, 3, 3], dtype=uint8)
```

`b` is of type `np.memmap` but is actually backed by an in-memory buffer and not a memmaped file. The fact that `b`'s data is in-memmory is fine. But I think `b` should just be a regular number array.

Anyway this is not directly the topic of this issue. I am +1 for a deprecation cycle to change the behavior of average and other aggregation operations.

Re ufuncs on memmaps:

Maybe we just need to add

``` python
    def __array_wrap__(self, arr, context=None):
        if arr.shape == ():
            return arr[()]
        return arr.view(np.ndarray)
```

or similar to the memmap definition. I just tried it, and it seems to work without failing any tests.

I think this is a good idea (unless there are ufuncs that do inplace operations?).

Also to fix the `memmap` class consistency issue I had prototyped in the past an alternative that would not subclass ndarray but instead call `np.frombuffer` on a custom mmap buffer that would store the required metadata (filename, offset, mode...):

https://github.com/joblib/joblib/commit/b68a86891f6a90dfe74043e59cfb9b5f50b738de#diff-19004d02cec64e3d448b33565de338ebR94

I never finished that work but if people are interested I could contribute it to numpy as a saner alternative to `np.memmap`.

@yarikoptic Could you add some specific examples of what you want to see?

oh boy -- I do not remember details NOW. is there a backup of those issues in their entirety somewhere to refresh what caused me to file the issue? (urls now get redirected here to github)

If there were any attachments, they would be here along with any comments. But I agree that this is ancient, about 1000 years in computer seconds ;) If you can't jog your memory, I'll close this because I don't know what it is talking about. Not that I've got much experience with memmapped files in the first place. 

I can't access the original ticket, but in NumPy 1.8.2, `x.sum()` gives a scalar if `x` is a regular `ndarray`, and a zero-dimensional `memmap` if `x` is a `memmap` instance.  As a result, `np.isscalar(x.sum())` passes in the first case and fails in the second (and as I recently discovered, `scipy.ndimage.measurements.center_of_mass` breaks on `memmap` instances because of this).

This discrepancy _seems_ to fit with numpy-gitbot's original message for this issue.

> as I recently discovered

Hmm; that's overstating my involvement: this came from investigating someone else's Stack Overflow question.  See here: http://stackoverflow.com/q/25369982/270986

The sum method is implemented with `add.reduce`, see `numpy/core/_methods.py`. If that is the problem, I would suspect that `x.min` and friends should also cause problems.

Your suspicions seem to be accurate :grinning:.

```
Python 3.4.1 (default, May 21 2014, 01:39:38) 
[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.__version__
'1.8.2'
>>> x = np.memmap("my_memmap", dtype=float, mode='w+', shape=(3, 4))
>>> x[...] = np.arange(12).reshape((3, 4))
>>> x.sum()
memmap(66.0)
>>> np.array(x).sum()
66.0
>>> x.min()
memmap(0.0)
>>> np.array(x).min()
0.0
```

Hmm, by and large, the operations try to preserve subclasses, and memmap inherits from ndarray. The best fix for this might be in numpy.memmap, defined in `numpy/core/memmap.py`, but it isn't clear to me how. Just making memmap scalars numpy scalars might be a bit tricky. They should be an instances of ndarray in any case. Might be that the `isscalar` function could be fixed to recognize scalar instances of subclasses of ndarray. 

For instance,

```
In [5]: isscalar(1)
Out[5]: True

In [6]: class Foo(int): pass

In [7]: isscalar(Foo(1))
Out[7]: False
```

If people are going to subclass ndarray and scalar types, we might want to modify `isscalar`.

That would be a bit larger modification though. 0-d arrays (array scalars) are currently not considered scalars by isscalar. To me the subclass thing seems like broadening it is good though.

is it possible to use `__array_wrap__` or something similar in memmap.py to return a ndarray if the result is not a view on a memmap ?


Resolving patch:
diff --git a/numpy/core/memmap.py b/numpy/core/memmap.py
--- a/numpy/core/memmap.py
+++ b/numpy/core/memmap.py
@@ -309,3 +309,24 @@ def flush(self):
         """
         if self.base is not None and hasattr(self.base, 'flush'):
             self.base.flush()
+
+    def __array_wrap__(self, arr, context=None):
+        arr = super(memmap, self).__array_wrap__(arr, context)
+
+        # Return a memmap if a memmap was given as the output of the
+        # ufunc. Leave the arr class unchanged if self is not a memmap
+        # to keep original memmap subclasses behavior
+        if self is arr or type(self) is not memmap:
+            return arr
+        # Return scalar instead of 0d memmap, e.g. for np.sum with
+        # axis=None
+        if arr.shape == ():
+            return arr[()]
+        # Return ndarray otherwise
+        return arr.view(np.ndarray)
+
+    def __getitem__(self, index):
+        res = super(memmap, self).__getitem__(index)
+        if type(res) is memmap and res._mmap is None:
+            return res.view(type=ndarray)
+        return res


Test patch:
diff --git a/numpy/core/tests/test_memmap.py b/numpy/core/tests/test_memmap.py
--- a/numpy/core/tests/test_memmap.py
+++ b/numpy/core/tests/test_memmap.py
@@ -5,7 +5,9 @@
 import shutil
 from tempfile import NamedTemporaryFile, TemporaryFile, mktemp, mkdtemp
 
-from numpy import memmap
+from numpy import (
+    memmap, sum, average, product, ndarray, isscalar, add, subtract, multiply)
+
 from numpy import arange, allclose, asarray
 from numpy.testing import (
     TestCase, run_module_suite, assert_, assert_equal, assert_array_equal,
@@ -126,5 +128,50 @@ def test_view(self):
         new_array = asarray(fp)
         assert_(new_array.base is fp)
 
+    def test_ufunc_return_ndarray(self):
+        fp = memmap(self.tmpfp, dtype=self.dtype, shape=self.shape)
+        fp[:] = self.data
+
+        for unary_op in [sum, average, product]:
+            result = unary_op(fp)
+            assert_(isscalar(result))
+            assert_(result.__class__ is self.data[0, 0].__class__)
+
+            assert_(unary_op(fp, axis=0).__class__ is ndarray)
+            assert_(unary_op(fp, axis=1).__class__ is ndarray)
+
+        for binary_op in [add, subtract, multiply]:
+            assert_(binary_op(fp, self.data).__class__ is ndarray)
+            assert_(binary_op(self.data, fp).__class__ is ndarray)
+            assert_(binary_op(fp, fp).__class__ is ndarray)
+
+        fp += 1
+        assert(fp.__class__ is memmap)
+        add(fp, 1, out=fp)
+        assert(fp.__class__ is memmap)
+
+    def test_getitem(self):
+        fp = memmap(self.tmpfp, dtype=self.dtype, shape=self.shape)
+        fp[:] = self.data
+
+        assert_(fp[1:, :-1].__class__ is memmap)
+        # Fancy indexing returns a copy that is not memmapped
+        assert_(fp[[0, 1]].__class__ is ndarray)
+
+    def test_memmap_subclass(self):
+        class MemmapSubClass(memmap):
+            pass
+
+        fp = MemmapSubClass(self.tmpfp, dtype=self.dtype, shape=self.shape)
+        fp[:] = self.data
+
+        # We keep previous behavior for subclasses of memmap, i.e. the
+        # ufunc and __getitem__ output is never turned into a ndarray
+        assert_(sum(fp, axis=0).__class__ is MemmapSubClass)
+        assert_(sum(fp).__class__ is MemmapSubClass)
+        assert_(fp[1:, :-1].__class__ is MemmapSubClass)
+        assert(fp[[0, 1]].__class__ is MemmapSubClass)
+
+
 if __name__ == "__main__":
     run_module_suite()