Repository: dask/dask

Issue:
da.cache
Should we provide a mechanism to cache dask.arrays?  This would compute and store the array and then create a new Array with a fresh graph.

``` python
def cache(self, storage=None):
    """ Return new dask.array with empty graph that points to evaluated array """
    if storage is None:
        storage = ... # some temporary on-disk store
    self.store(storage)
    return da.from_array(storage, chunks=self.chunks)
```

This could be improved by the following:
1.  Figuring out how to do a nice temporary on-disk array store
2.  Supporting multiple inputs/stores at once



Discussion:
So we would be saving the _graph_ generated by daks.array? Not a numpy array?

So as a first approach, saving would  be like this?

``` python
def save(self):
    with open(file_name) as f:
        data = pickle.dumps((self.dask, self.keys, self.chunks))
        f.write(data)
```

The original idea would be that we would be to trigger computation up to this point, store the result in an array, and then create a new dask.array from that array with a fresh, fairly empty graph.

This is also a great idea. HDF5 is the obvious storage mechanism, but it's out because it only does homogeneously sized chunks.

I would use a hybrid format that stores each chunk in a single binary blob and uses a JSON file to keep track of the index. Optionally, you can compress each chunk (e.g., with gzip or blosc). If you want it to be fully portable, I would store raw bytes (numpy.tostring instead of pickle) in each of the chunk files and put all the shape/dtype metadata in the JSON. 

With the example above we would just use anything that supports the numpy setitem syntax.  HDF5 would do ok, it can handle concurrent writes safely.  We're not being optimally efficient, but that might be ok.

In dask.dataframe.cache we do something more custom like what @shoyer suggests, we dump the values straight into a `chest` without storing metadata in the temporary store.  We copy the metadata directly over to the new dask collection object without serializing it anywhere.

Yeah, actually using chest would be a nice simple way to handle this.

On Wed, Jun 10, 2015 at 4:36 PM, Matthew Rocklin notifications@github.com
wrote:

> With the example above we would just use anything that supports the numpy setitem syntax.  HDF5 would do ok, it can handle concurrent writes safely.  We're not being optimally efficient, but that might be ok.
> 
> ## In dask.dataframe.cache we do something more custom like what @shoyer suggests, we dump the values straight into a `chest` without storing metadata in the temporary store.  We copy the metadata directly over to the new dask collection object without serializing it anywhere.
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/ContinuumIO/dask/issues/293#issuecomment-110921748

Couldn't this also be a way to implement the eager evaluation mode of issue #292 ?

> Couldn't this also be a way to implement the eager evaluation mode of issue #292 ?

Yes.  That sounds like a great idea.


Resolving patch:
diff --git a/dask/array/core.py b/dask/array/core.py
--- a/dask/array/core.py
+++ b/dask/array/core.py
@@ -4,7 +4,7 @@
 from operator import add, getitem
 import inspect
 from numbers import Number
-from collections import Iterable
+from collections import Iterable, MutableMapping
 from bisect import bisect
 from itertools import product, count
 from collections import Iterator
@@ -716,6 +716,77 @@ def compute(self, **kwargs):
         result, = compute(self, **kwargs)
         return result
 
+    def cache(self, store=None, **kwargs):
+        """ Evaluate and cache array
+
+        Parameters
+        ----------
+        store: MutableMapping or ndarray-like
+            Place to put computed and cached chunks
+        kwargs:
+            Keyword arguments to pass on to ``get`` function for scheduling
+
+        This triggers evaluation and store the result in either
+
+        1.  An ndarray object supporting setitem (see da.store)
+        2.  A MutableMapping like a dict or chest
+
+        It then returns a new dask array that points to this store.
+        This returns a semantically equivalent dask array.
+
+        >>> import dask.array as da
+        >>> x = da.arange(5, chunks=2)
+        >>> y = 2*x + 1
+        >>> z = y.cache()  # triggers computation
+
+        >>> y.compute()  # Does entire computation
+        array([1, 3, 5, 7, 9])
+
+        >>> z.compute()  # Just pulls from store
+        array([1, 3, 5, 7, 9])
+
+        You might base a cache off of an array like a numpy array or
+        h5py.Dataset.
+
+        >>> cache = np.empty(5, dtype=x.dtype)
+        >>> z = y.cache(store=cache)
+        >>> cache
+        array([1, 3, 5, 7, 9])
+
+        Or one might use a MutableMapping like a dict or chest
+
+        >>> cache = dict()
+        >>> z = y.cache(store=cache)
+        >>> cache  # doctest: +SKIP
+        {('x', 0): array([1, 3]),
+         ('x', 1): array([5, 7]),
+         ('x', 2): array([9])}
+        """
+        if store is not None and hasattr(store, 'shape'):
+            self.store(store)
+            return from_array(store, chunks=self.chunks)
+        if store is None:
+            try:
+                from chest import Chest
+                store = Chest()
+            except ImportError:
+                if self.nbytes <= 1e9:
+                    store = dict()
+                else:
+                    raise ValueError("No out-of-core storage found."
+                        "Either:\n"
+                        "1. Install ``chest``, an out-of-core dictionary\n"
+                        "2. Provide an on-disk array like an h5py.Dataset") # pragma: no cover
+        if isinstance(store, MutableMapping):
+            name = next(names)
+            dsk = dict(((name, k[1:]), (operator.setitem, store, (tuple, list(k)), k))
+                    for k in core.flatten(self._keys()))
+            get(merge(dsk, self.dask), list(dsk.keys()), **kwargs)
+
+            dsk2 = dict((k, (operator.getitem, store, (tuple, list(k))))
+                        for k in store)
+            return Array(dsk2, self.name, chunks=self.chunks, dtype=self._dtype)
+
     def __int__(self):
         return int(self.compute())
     def __bool__(self):


Test patch:
diff --git a/dask/array/tests/test_array_core.py b/dask/array/tests/test_array_core.py
--- a/dask/array/tests/test_array_core.py
+++ b/dask/array/tests/test_array_core.py
@@ -1029,3 +1029,17 @@ def test_numpy_compat_is_notimplemented():
     x = da.from_array(a, chunks=5)
 
     assert raises(NotImplementedError, lambda: x + a)
+
+
+def test_cache():
+    x = da.arange(15, chunks=5)
+    y = 2 * x + 1
+    z = y.cache()
+    assert len(z.dask) == 3  # very short graph
+    assert eq(y, z)
+
+    cache = np.empty(15, dtype=y.dtype)
+    z = y.cache(store=cache)
+    assert len(z.dask) < 6  # very short graph
+    assert z.chunks == y.chunks
+    assert eq(y, z)