Repository: pydata/sparse

Issue:
Disallow np.array(COO)
Currently, we have a mix of when to allow dense inputs and when not to allow them. Currently, we have the `__array__` protocol which converts the sparse array into a dense one when using `np.array(COO)` or similar.

This would be fine if we knew that we were densifying, but in practice, we really don't. Think of something that calls `np.asarray` internally, this would densify, and in many cases fill up memory and raise a `MemoryError`.

In practice, what we want most of the time is to disallow `np.[as]array(COO)` and provide an _explicit_ escape hatch in the form of `COO.todense()`.

Recently, I added an environment variable to control this, so this discussion is mostly about changing the default behaviour.

Thoughts, @rgommers, @shoyer, @mrocklin, @perimosocordiae?

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/pydata/sparse/pull/220?src=pr&el=h1) Report
> Merging [#220](https://codecov.io/gh/pydata/sparse/pull/220?src=pr&el=desc) into [master](https://codecov.io/gh/pydata/sparse/commit/e46a2a734dce1b25a775641f744cbcbea1a15910?src=pr&el=desc) will **decrease** coverage by `0.25%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pydata/sparse/pull/220/graphs/tree.svg?width=650&token=H212u0Uxxw&height=150&src=pr)](https://codecov.io/gh/pydata/sparse/pull/220?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #220      +/-   ##
==========================================
- Coverage   97.59%   97.33%   -0.26%     
==========================================
  Files          11       11              
  Lines        1539     1541       +2     
==========================================
- Hits         1502     1500       -2     
- Misses         37       41       +4
```


| [Impacted Files](https://codecov.io/gh/pydata/sparse/pull/220?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sparse/coo/core.py](https://codecov.io/gh/pydata/sparse/pull/220/diff?src=pr&el=tree#diff-c3BhcnNlL2Nvby9jb3JlLnB5) | `96.66% <ø> (-0.05%)` | :arrow_down: |
| [sparse/sparse\_array.py](https://codecov.io/gh/pydata/sparse/pull/220/diff?src=pr&el=tree#diff-c3BhcnNlL3NwYXJzZV9hcnJheS5weQ==) | `97.36% <100%> (+0.59%)` | :arrow_up: |
| [sparse/compatibility.py](https://codecov.io/gh/pydata/sparse/pull/220/diff?src=pr&el=tree#diff-c3BhcnNlL2NvbXBhdGliaWxpdHkucHk=) | `50% <0%> (-33.34%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pydata/sparse/pull/220?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pydata/sparse/pull/220?src=pr&el=footer). Last update [e46a2a7...cf549c5](https://codecov.io/gh/pydata/sparse/pull/220?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

scopatz: LGTM! Thanks @hameerabbasi!

Patch:
diff --git a/docs/generated/sparse.SparseArray.rst b/docs/generated/sparse.SparseArray.rst
index 031e918..dfa07dd 100644
--- a/docs/generated/sparse.SparseArray.rst
+++ b/docs/generated/sparse.SparseArray.rst
@@ -19,4 +19,5 @@ SparseArray
       :toctree:
 
       SparseArray.asformat
+      SparseArray.todense
 
diff --git a/docs/generated/sparse.SparseArray.todense.rst b/docs/generated/sparse.SparseArray.todense.rst
new file mode 100644
index 0000000..b538174
--- /dev/null
+++ b/docs/generated/sparse.SparseArray.todense.rst
@@ -0,0 +1,6 @@
+SparseArray.todense
+===================
+
+.. currentmodule:: sparse
+
+.. automethod:: SparseArray.todense
\ No newline at end of file
diff --git a/sparse/__init__.py b/sparse/__init__.py
index 7b1186f..986371f 100644
--- a/sparse/__init__.py
+++ b/sparse/__init__.py
@@ -10,7 +10,7 @@ del get_versions
 
 import os
 
-_AUTO_DENSIFICATION_ENABLED = bool(int(os.environ.get('SPARSE_AUTO_DENSIFY', '1')))
+_AUTO_DENSIFICATION_ENABLED = bool(int(os.environ.get('SPARSE_AUTO_DENSIFY', '0')))
 _AUTO_WARN_ON_TOO_DENSE = bool(int(os.environ.get('SPARSE_WARN_ON_TOO_DENSE', '0')))
 
 del os
\ No newline at end of file
diff --git a/sparse/coo/common.py b/sparse/coo/common.py
index f189f58..83da272 100644
--- a/sparse/coo/common.py
+++ b/sparse/coo/common.py
@@ -612,7 +612,7 @@ def nanmean(x, axis=None, keepdims=False, dtype=None, out=None):
     den = reduce(operator.mul, (x.shape[i] for i in axis), 1)
     den -= nancount
 
-    if np.any(den == 0):
+    if (den == 0).any():
         warnings.warn("Mean of empty slice", RuntimeWarning, stacklevel=2)
 
     num = np.sum(x2, axis=axis, dtype=dtype, keepdims=keepdims)
diff --git a/sparse/coo/core.py b/sparse/coo/core.py
index 768e6e8..c62797a 100644
--- a/sparse/coo/core.py
+++ b/sparse/coo/core.py
@@ -1449,14 +1449,6 @@ class COO(SparseArray, NDArrayOperatorsMixin):
 
         return result
 
-    def __array__(self, **kwargs):
-        from .. import _AUTO_DENSIFICATION_ENABLED as densify
-        if not densify:
-            raise RuntimeError('Cannot convert a sparse array to dense automatically. '
-                               'To manually densify, use the todense method.')
-
-        return np.asarray(self.todense(), **kwargs)
-
     def linear_loc(self):
         """
         The nonzero coordinates of a flattened version of this array. Note that
diff --git a/sparse/sparse_array.py b/sparse/sparse_array.py
index 97fc37c..599d289 100644
--- a/sparse/sparse_array.py
+++ b/sparse/sparse_array.py
@@ -5,6 +5,8 @@ from numbers import Integral
 from functools import reduce
 import operator
 
+import numpy as np
+
 from .utils import _zero_of_dtype
 from .compatibility import int
 
@@ -177,3 +179,38 @@ class SparseArray(object):
         NotImplementedError
             If the format isn't supported.
         """
+
+    @abstractmethod
+    def todense(self):
+        """
+        Convert this :obj:`SparseArray` array to a dense :obj:`numpy.ndarray`. Note that
+        this may take a large amount of memory and time.
+
+        Returns
+        -------
+        numpy.ndarray
+            The converted dense array.
+
+        See Also
+        --------
+        DOK.todense : Equivalent :obj:`DOK` array method.
+        COO.todense : Equivalent :obj:`COO` array method.
+        scipy.sparse.coo_matrix.todense : Equivalent Scipy method.
+
+        Examples
+        --------
+        >>> import sparse
+        >>> x = np.random.randint(100, size=(7, 3))
+        >>> s = sparse.COO.from_numpy(x)
+        >>> x2 = s.todense()
+        >>> np.array_equal(x, x2)
+        True
+        """
+
+    def __array__(self, **kwargs):
+        from . import _AUTO_DENSIFICATION_ENABLED as densify
+        if not densify:
+            raise RuntimeError('Cannot convert a sparse array to dense automatically. '
+                               'To manually densify, use the todense method.')
+
+        return np.asarray(self.todense(), **kwargs)


Test patch:
diff --git a/sparse/tests/test_coo.py b/sparse/tests/test_coo.py
index 79c70b6..549f3eb 100644
--- a/sparse/tests/test_coo.py
+++ b/sparse/tests/test_coo.py
@@ -1645,9 +1645,9 @@ def test_size():
 
 def test_np_array():
     s = sparse.random((20, 30, 40))
-    x = np.array(s)
-    assert isinstance(x, np.ndarray)
-    assert_eq(x, s)
+
+    with pytest.raises(RuntimeError):
+        np.array(s)
 
 
 @pytest.mark.parametrize('shapes', [
@@ -2059,12 +2059,14 @@ def test_failed_densification():
     import os
     from importlib import reload
 
-    os.environ['SPARSE_AUTO_DENSIFY'] = '0'
+    os.environ['SPARSE_AUTO_DENSIFY'] = '1'
     reload(sparse)
 
-    with pytest.raises(RuntimeError):
-        s = sparse.random((3, 4, 5), density=0.5)
-        np.array(s)
+    s = sparse.random((3, 4, 5), density=0.5)
+    x = np.array(s)
+
+    assert isinstance(x, np.ndarray)
+    assert_eq(s, x)
 
     del os.environ['SPARSE_AUTO_DENSIFY']
     reload(sparse)