Repository: dask/dask

Issue:
Convert dask.imperative values into dask collections
Dask imperative transparently converts dask collections into `Value` objects. It would be useful to be able to do the opposite -- that is, to convert a lazy value into a dask array (or dataframe).

This would be quite useful for constructing dask collections from arbitrary user defined code, as a more flexible alternative to `da.core.from_func` (https://github.com/blaze/dask/pull/400).

Here's an example, based on a use case that came up last month and for which I had to write a custom dask graph:

``` python
from dask.imperative import do
import dask.array as da

def data_for_one_day(time):
    # given a datetime, downloads the relevant data from the cloud and returns
    # a big 2d numpy array and an error code extracted from the metadata
    return data, error_code

results = [do(data_for_one_day)(time) for time in times]
arrays = da.stack([da.from_imperative(r[0], (3000, 5000), 'float32') for r in results])
errors = da.stack([da.from_imperative(r[1], (), 'int64') for r in results])
```

The basics of this function look something like the following:

``` python
def from_imperative(val, shape, dtype, name):
    dsk = {(name,) + (0,) * len(shape): val.key}
    dsk.update(val.dask)
    return Array(dsk, name, ((d,) for d in shape), dtype)
```

I kind of like the idea of reusing the generic `da.array` or `da.asarray` functions for generic coercion into dask array objects, though I can see how this could be confusing (we'd also have to add a `shape` argument to the numpy signature).



Discussion:

Resolving patch:
diff --git a/dask/array/__init__.py b/dask/array/__init__.py
--- a/dask/array/__init__.py
+++ b/dask/array/__init__.py
@@ -2,10 +2,10 @@
 
 from ..utils import ignoring
 from .core import (Array, stack, concatenate, take, tensordot, transpose,
-        from_array, choose, where, coarsen, insert, broadcast_to, ravel,
-        reshape, fromfunction, unique, store, squeeze, topk, bincount,
-        histogram, map_blocks, atop, to_hdf5, dot, cov, array, to_npy_stack,
-        from_npy_stack)
+        from_array, from_imperative, choose, where, coarsen, insert,
+        broadcast_to, ravel, reshape, fromfunction, unique, store, squeeze,
+        topk, bincount, histogram, map_blocks, atop, to_hdf5, dot, cov, array,
+        to_npy_stack, from_npy_stack)
 from .core import (logaddexp, logaddexp2, conj, exp, log, log2, log10, log1p,
         expm1, sqrt, square, sin, cos, tan, arcsin, arccos, arctan, arctan2,
         hypot, sinh, cosh, tanh, arcsinh, arccosh, arctanh, deg2rad, rad2deg,
diff --git a/dask/array/core.py b/dask/array/core.py
--- a/dask/array/core.py
+++ b/dask/array/core.py
@@ -1227,6 +1227,33 @@ def from_array(x, chunks, name=None, lock=False):
     return Array(merge({name: x}, dsk), name, chunks, dtype=x.dtype)
 
 
+def from_imperative(value, shape, dtype=None, name=None):
+    """ Create a dask array from a dask imperative value
+
+    This routine is useful for constructing dask arrays in an ad-hoc fashion
+    using dask imperative, particularly when combined with stack and
+    concatenate.
+
+    The dask array will consist of a single chunk.
+
+    Examples
+    --------
+
+    >>> from dask import do
+    >>> value = do(np.ones)(5)
+    >>> array = from_imperative(value, (5,), dtype=float)
+    >>> array
+    dask.array<from-va..., shape=(5,), dtype=float64, chunksize=(5,)>
+    >>> array.compute()
+    array([ 1.,  1.,  1.,  1.,  1.])
+    """
+    name = name or 'from-value-' + tokenize(value, shape, dtype)
+    dsk = {(name,) + (0,) * len(shape): value.key}
+    dsk.update(value.dask)
+    chunks = tuple((d,) for d in shape)
+    return Array(dsk, name, chunks, dtype)
+
+
 def from_func(func, shape, dtype=None, name=None, args=(), kwargs={}):
     """ Create dask array in a single block by calling a function
 


Test patch: