Repository: numpy/numpy

Issue:
np.average of memmap returns memmap 0d array instead of numpy scalar
``` python
import numpy as np
mm = np.memmap('/tmp/test.data', mode='w+', shape=4)
np.average(mm)
```

Output since 5ceab8f982d80c68b3b5e90697cd20df51f4a3e8 (a couple of days ago):

```
memmap(0.0)
```

Output before that (e.g. in 1.10.4):

```
0.0
```

This causes the scikit-learn using numpy master tests to fail, see this Travis [log](https://travis-ci.org/scikit-learn/scikit-learn/jobs/114973679).

We have been bitten in the past by this in scikit-learn with `.sum` that has the same surprising behaviour (see https://github.com/scikit-learn/scikit-learn/pull/6225). Shouldn't reduce operations on a memmap array always return a numpy scalar ?

cc @ogrisel.

np.average of memmap returns memmap 0d array instead of numpy scalar
``` python
import numpy as np
mm = np.memmap('/tmp/test.data', mode='w+', shape=4)
np.average(mm)
```

Output since 5ceab8f982d80c68b3b5e90697cd20df51f4a3e8 (a couple of days ago):

```
memmap(0.0)
```

Output before that (e.g. in 1.10.4):

```
0.0
```

This causes the scikit-learn using numpy master tests to fail, see this Travis [log](https://travis-ci.org/scikit-learn/scikit-learn/jobs/114973679).

We have been bitten in the past by this in scikit-learn with `.sum` that has the same surprising behaviour (see https://github.com/scikit-learn/scikit-learn/pull/6225). Shouldn't reduce operations on a memmap array always return a numpy scalar ?

cc @ogrisel.



Discussion:
The cause is the change from `a = np.asarray(a)` to `a = np.asanyarray(a)` at the start of `np.average`, which I did to preserve subclass like `np.mean` does. Previously `np.average` only preserved the `np.matrix` subclass using a special clause, but now it preserves any subclass.

While it's definitely a change in behavior we should think carefully about, the new behavior seems more consistent with the rest of numpy. If no weights are given `np.average` just computes `np.mean`, and `np.mean` preserves the subclass (including for scalars), just like ufuncs and most other numpy methods. This is so that subclasses have a chance to modify the inputs or outputs to the computation, see some of the discussion in #5819, #667.

I suggest we should keep the old `asarray` behavior temporarily with a deprecation warning, and then (sorry @lesteve) go ahead with the change to `asanyarray` in the next release. Is that reasonable?

Possibly we might also think about updating `memmap.sum` and others so they return a scalar, or making some of the changes proposed in #667. Indeed it seems weird that memmap sum/mean return a memmap array (even for non-scalar results, eg after summing over an axis).

Pretty much all the operations are weird anyway with `numpy.memmap`, for instance the product of a memmap array by a scalar is a memmap array too: 

``` python
>>> import numpy as np
>>> a = np.memmap('a.bin', shape=10, dtype=np.uint8, mode='w+')
>>> a[:] = 1

>>> b = a * 3
>>> type(b)
<class 'numpy.core.memmap.memmap'>
>>> b.base
array([3, 3, 3, 3, 3, 3, 3, 3, 3, 3], dtype=uint8)
```

`b` is of type `np.memmap` but is actually backed by an in-memory buffer and not a memmaped file. The fact that `b`'s data is in-memmory is fine. But I think `b` should just be a regular number array.

Anyway this is not directly the topic of this issue. I am +1 for a deprecation cycle to change the behavior of average and other aggregation operations.

Re ufuncs on memmaps:

Maybe we just need to add

``` python
    def __array_wrap__(self, arr, context=None):
        if arr.shape == ():
            return arr[()]
        return arr.view(np.ndarray)
```

or similar to the memmap definition. I just tried it, and it seems to work without failing any tests.

I think this is a good idea (unless there are ufuncs that do inplace operations?).

Also to fix the `memmap` class consistency issue I had prototyped in the past an alternative that would not subclass ndarray but instead call `np.frombuffer` on a custom mmap buffer that would store the required metadata (filename, offset, mode...):

https://github.com/joblib/joblib/commit/b68a86891f6a90dfe74043e59cfb9b5f50b738de#diff-19004d02cec64e3d448b33565de338ebR94

I never finished that work but if people are interested I could contribute it to numpy as a saner alternative to `np.memmap`.

The cause is the change from `a = np.asarray(a)` to `a = np.asanyarray(a)` at the start of `np.average`, which I did to preserve subclass like `np.mean` does. Previously `np.average` only preserved the `np.matrix` subclass using a special clause, but now it preserves any subclass.

While it's definitely a change in behavior we should think carefully about, the new behavior seems more consistent with the rest of numpy. If no weights are given `np.average` just computes `np.mean`, and `np.mean` preserves the subclass (including for scalars), just like ufuncs and most other numpy methods. This is so that subclasses have a chance to modify the inputs or outputs to the computation, see some of the discussion in #5819, #667.

I suggest we should keep the old `asarray` behavior temporarily with a deprecation warning, and then (sorry @lesteve) go ahead with the change to `asanyarray` in the next release. Is that reasonable?

Possibly we might also think about updating `memmap.sum` and others so they return a scalar, or making some of the changes proposed in #667. Indeed it seems weird that memmap sum/mean return a memmap array (even for non-scalar results, eg after summing over an axis).

Pretty much all the operations are weird anyway with `numpy.memmap`, for instance the product of a memmap array by a scalar is a memmap array too: 

``` python
>>> import numpy as np
>>> a = np.memmap('a.bin', shape=10, dtype=np.uint8, mode='w+')
>>> a[:] = 1

>>> b = a * 3
>>> type(b)
<class 'numpy.core.memmap.memmap'>
>>> b.base
array([3, 3, 3, 3, 3, 3, 3, 3, 3, 3], dtype=uint8)
```

`b` is of type `np.memmap` but is actually backed by an in-memory buffer and not a memmaped file. The fact that `b`'s data is in-memmory is fine. But I think `b` should just be a regular number array.

Anyway this is not directly the topic of this issue. I am +1 for a deprecation cycle to change the behavior of average and other aggregation operations.

Re ufuncs on memmaps:

Maybe we just need to add

``` python
    def __array_wrap__(self, arr, context=None):
        if arr.shape == ():
            return arr[()]
        return arr.view(np.ndarray)
```

or similar to the memmap definition. I just tried it, and it seems to work without failing any tests.

I think this is a good idea (unless there are ufuncs that do inplace operations?).

Also to fix the `memmap` class consistency issue I had prototyped in the past an alternative that would not subclass ndarray but instead call `np.frombuffer` on a custom mmap buffer that would store the required metadata (filename, offset, mode...):

https://github.com/joblib/joblib/commit/b68a86891f6a90dfe74043e59cfb9b5f50b738de#diff-19004d02cec64e3d448b33565de338ebR94

I never finished that work but if people are interested I could contribute it to numpy as a saner alternative to `np.memmap`.


Resolving patch:
diff --git a/numpy/lib/function_base.py b/numpy/lib/function_base.py
--- a/numpy/lib/function_base.py
+++ b/numpy/lib/function_base.py
@@ -1002,7 +1002,19 @@ def average(a, axis=None, weights=None, returned=False):
     TypeError: Axis must be specified when shapes of a and weights differ.
 
     """
-    a = np.asanyarray(a)
+    # 3/19/2016 1.12.0:
+    # replace the next few lines with "a = np.asanyarray(a)"
+    if (type(a) not in (np.ndarray, np.matrix) and
+            issubclass(type(a), np.ndarray)):
+        warnings.warn("np.average currently does not preserve subclasses, but "
+                      "will do so in the future to match the behavior of most "
+                      "other numpy functions such as np.mean. In particular, "
+                      "this means calls which returned a scalar may return a "
+                      "0-d subclass object instead.",
+                      FutureWarning, stacklevel=2)
+
+    if not isinstance(a, np.matrix):
+        a = np.asarray(a)
 
     if weights is None:
         avg = a.mean(axis)


Test patch: