Repository: dask/dask

Issue:
Memory issue when numpy arrays are passed to dask arrays
I have a 68gb ram cluster running which runs into memory issues (which kills the worker) only when Numpy arrays are passed to Dask arrays.

![image](https://user-images.githubusercontent.com/29863388/69137246-a6652f80-0abc-11ea-80ab-ef3f15789f6f.png)

 Dask array reproducible example:

```py
import dask.array as da
import numpy as np
from dask.distributed import Client


client = Client()

x = da.random.uniform(low=0, high=10, size=(112_266_893, 15))
x.std(axis=0).compute()  # works just fine

```

However, I am trying to pass a numpy array (representing my feature matrix with the same size as the simulated dask array. I ran into memory issues when I pass the numpy array to dask array.

Numpy array reproducible example:

```py
test_np = np.random.uniform(low=0, high=10, size=(112_266_893, 15))
test_da = da.from_array(test_np)

test_da.mean(axis=0).compute()  # fails after worker gets killed
```

![image](https://user-images.githubusercontent.com/29863388/69137877-e5e04b80-0abd-11ea-9227-26726fbf1bb3.png)


My question is why does dask array compute work in the former but fail in the latter?

Here's the console output where dask seems to be reporting a potential memory leak from this example:

![image](https://user-images.githubusercontent.com/29863388/69137775-b3365300-0abd-11ea-8537-a1ec299e2c74.png)
 


Am I doing something wrong, or do I just need more ram?





Discussion:
Maybe your numpy array is to large to safely put on a single worker?  When
you use Dask arrays they are broken up into many small pieces so that you
can solve larger problems than you have ram.  When you use numpy, numpy
allocates a single array which has to fit in ram.

See https://docs.dask.org/en/latest/best-practices.html#load-data-with-dask

On Tue, Nov 19, 2019 at 3:15 AM Bernard Brenyah <notifications@github.com>
wrote:

> I have a 68gb ram cluster running which runs into memory issues (which
> kills the worker) only when Numpy arrays are passed to Dask arrays.
>
> [image: image]
> <https://user-images.githubusercontent.com/29863388/69137246-a6652f80-0abc-11ea-80ab-ef3f15789f6f.png>
>
> Dask array reproducible example:
>
> import dask.array as daimport numpy as npfrom dask.distributed import Client
>
>
> client = Client()
>
> x = da.random.uniform(low=0, high=10, size=(112_266_893, 15))
> x.std(axis=0).compute()  # works just fine
>
> However, I am trying to pass a numpy array (representing my feature matrix
> with the same size as the simulated dask array. I ran into memory issues
> when I pass the numpy array to dask array.
>
> Numpy array reproducible example:
>
> test_np = np.random.uniform(low=0, high=10, size=(112_266_893, 15))
> test_da = da.from_array(test_np)
>
> test_da.mean(axis=0).compute()  # fails after worker gets killed
>
> [image: image]
> <https://user-images.githubusercontent.com/29863388/69137877-e5e04b80-0abd-11ea-9227-26726fbf1bb3.png>
>
> My question is why does dask array compute work in the former but fail in
> the latter?
>
> Here's the console output where dask seems to be reporting a potential
> memory leak from this example:
>
> [image: image]
> <https://user-images.githubusercontent.com/29863388/69137775-b3365300-0abd-11ea-8537-a1ec299e2c74.png>
>
> Am I doing something wrong, or do I just need more ram?
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/dask/issues/5601?email_source=notifications&email_token=AACKZTGT3E3WHOF7HPAYTZDQUO4FLA5CNFSM4JPA76RKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4H2JPJEA>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACKZTHFN2LPSG4B3BF3EADQUO4FLANCNFSM4JPA76RA>
> .
>

@PyDataBlog you gave a thumbs up. Does that mean Matt's comment and the docs explain things adequately?
This looks like it is the same problem I reported over in distributed: https://github.com/dask/distributed/issues/3032.

If you call `da.from_array(numpy_array)`, regardless of the chunking of the dask array, any computation of any chunk will require the full array (as opposed to only requiring a chunk). For big arrays, this easily blows up a single worker.

So, If you actually want the numpy array to be divided into chunks, you should slice your array into sub-arrays and concatenate / stack them together using something like `da.stack([da.from_array(sub_array) for sub_array in sliced_numpy_array])`
@TomAugspurger I will try the suggestion from @d-v-b and report as well. 

@d-v-b how can I make  `sliced_numpy_array` chunk generator?
@PyDataBlog something like this:
```python
test_np = np.random.uniform(size=(112_266_893,15))
# window size
ws = 10000
# divide test_np into sub-arrays via simple slicing along the first dimension
sliced = [da.from_array(test_np[slice(ind * ws, (ind * ws) + ws )]) for ind in range(1 + test_np.shape[0]//ws)]
x = da.concatenate(sliced)
```
If you construct `x` like this, then the memory footprint of a single chunk should be `(ws, 15)` times the datatype. Whereas if you construct `x` just using `da.from_array(test_np)`, the memory footprint of a single chunk is the entire array.
@PyDataBlog did my suggestion work for you?
> @PyDataBlog did my suggestion work for you?

The client is now able to process that same test_np array object that it was failing earlier so that should solve the main task since they have the same shape and size. Feel free to close the issue. 
It's worth noting [`da.from_array(...)`]( https://docs.dask.org/en/latest/array-api.html#dask.array.from_array ) accepts a `chunks` argument as well.

Resolving patch:
diff --git a/dask/array/core.py b/dask/array/core.py
--- a/dask/array/core.py
+++ b/dask/array/core.py
@@ -2831,6 +2831,14 @@ def from_array(
     >>> import dask.base
     >>> token = dask.base.tokenize(x)  # doctest: +SKIP
     >>> a = da.from_array('myarray-' + token)  # doctest: +SKIP
+
+    Numpy ndarrays are eagerly sliced and then embedded in the graph.
+
+    >>> import dask.array
+    >>> a = dask.array.from_array(np.array([[1, 2], [3, 4]]), chunks=(1,1))
+    >>> a.dask[a.name, 0, 0][0]
+    array([1])
+
     """
     if isinstance(x, Array):
         raise ValueError(
@@ -2866,17 +2874,24 @@ def from_array(
     if lock is True:
         lock = SerializableLock()
 
+    is_ndarray = type(x) is np.ndarray
+    is_single_block = all(len(c) == 1 for c in chunks)
     # Always use the getter for h5py etc. Not using isinstance(x, np.ndarray)
     # because np.matrix is a subclass of np.ndarray.
-    if type(x) is np.ndarray and all(len(c) == 1 for c in chunks):
+    if is_ndarray and not is_single_block and not lock:
+        # eagerly slice numpy arrays to prevent memory blowup
+        # GH5367, GH5601
+        slices = slices_from_chunks(chunks)
+        keys = product([name], *(range(len(bds)) for bds in chunks))
+        values = [x[slc] for slc in slices]
+        dsk = dict(zip(keys, values))
+
+    elif is_ndarray and is_single_block:
         # No slicing needed
         dsk = {(name,) + (0,) * x.ndim: x}
     else:
         if getitem is None:
-            if type(x) is np.ndarray and not lock:
-                # simpler and cleaner, but missing all the nuances of getter
-                getitem = operator.getitem
-            elif fancy:
+            if fancy:
                 getitem = getter
             else:
                 getitem = getter_nofancy


Test patch:
diff --git a/dask/array/tests/test_array_core.py b/dask/array/tests/test_array_core.py
--- a/dask/array/tests/test_array_core.py
+++ b/dask/array/tests/test_array_core.py
@@ -2387,7 +2387,7 @@ def test_from_array_ndarray_getitem():
     x = np.array([[1, 2], [3, 4]])
     dx = da.from_array(x, chunks=(1, 2))
     assert_eq(x, dx)
-    assert dx.dask[dx.name, 0, 0][0] == operator.getitem
+    assert (dx.dask[dx.name, 0, 0] == np.array([[1, 2]])).all()
 
 
 @pytest.mark.parametrize("x", [[1, 2], (1, 2), memoryview(b"abc")])
@@ -2399,8 +2399,7 @@ def test_from_array_list(x):
 
     dx = da.from_array(x, chunks=1)
     assert_eq(np.array(x), dx)
-    assert dx.dask[dx.name, 0][0] == operator.getitem
-    assert isinstance(dx.dask[dx.name.replace("array", "array-original")], np.ndarray)
+    assert dx.dask[dx.name, 0][0] == x[0]
 
 
 @pytest.mark.parametrize("type_", [t for t in np.ScalarType if t is not memoryview])
@@ -3295,7 +3294,7 @@ def test_from_array_names():
     d = da.from_array(x, chunks=2)
 
     names = countby(key_split, d.dask)
-    assert set(names.values()) == set([1, 5])
+    assert set(names.values()) == set([5])
 
 
 @pytest.mark.parametrize(
diff --git a/dask/array/tests/test_optimization.py b/dask/array/tests/test_optimization.py
--- a/dask/array/tests/test_optimization.py
+++ b/dask/array/tests/test_optimization.py
@@ -264,7 +264,8 @@ def test_dont_fuse_numpy_arrays():
 
 
 def test_minimize_data_transfer():
-    x = np.ones(100)
+    zarr = pytest.importorskip("zarr")
+    x = zarr.ones((100,))
     y = da.from_array(x, chunks=25)
     z = y + 1
     dsk = z.__dask_optimize__(z.dask, z.__dask_keys__())