Repository: dask/dask

Issue:
Improve user experience of interacting with numpy arrays
Currently when numpy arrays interact with dask arrays we swallow errors silently, revealing them only at computation time.

``` python
In [1]: import dask.array as da
In [2]: import numpy as np
In [3]: x = da.arange(15, chunks=5)
In [4]: y = np.arange(15)

In [5]: x + y
Out[5]: dask.array<x_1, shape=(15,), chunks=((5, 5, 5)), dtype=int64>

In [6]: _.compute()
ValueError: operands could not be broadcast together with shapes (5,) (15,) 
```

We should do one of the following things:
1.  Figure out automatic conversion of a numpy array into a dask array with appropriate shape to make the operation work (hard in general, maybe easy in a few key cases)
2.  Raise an informative error early on (maybe easier in the hard cases)



Discussion:
We might be interested in adding a `__numpy_ufunc__` method. It lets third party objects override ufuncs. More info here 

https://github.com/numpy/numpy/blob/master/doc/neps/ufunc-overrides.rst

I worked on adding this to NumPy in 2013, and continuing development on it after the Summer became unfeasible because it there was a lot of maintenance and I was in school. It's MRO is currently being discussed here: 

https://github.com/numpy/numpy/issues/5844

I don't think the state of NumPy ufunc is mature or stable enough to warrant this yet.

For now I think a simple thing to do would be to raise a `NotImplementedError` immediately instead of waiting until compute time to fail.

@cowlicks  I think we'll definitely want to add `__numpy_ufunc__` once a NumPy release with it is out, if only so we don't need to write our own ufuncs like `dask.array.sin`. This will also integrate nicely with Numba's support for ufuncs on demand.

For the record, that thread is mostly about how to handle binary operation dispatching (e.g., `+`), and whether that should be independent from `__numpy_ufunc__` (e.g., `np.add`). I don't think anyone is taking issue with your design of `__numpy_ufunc__` :).

Probably this is hard to answer, but is there an ETA on a numpy release.

> Probably this is hard to answer, but is there an ETA on a numpy release.

Pretty sure it's supposed to come out this summer. I'm guessing as soon as this `__numpy_ufunc__` mess is resolved (or if it's deemed unresolvable for now). @charris is in charge.

For me this problem boils down to checking whether the inputs can broadcast to a common shape and if so, if the chunk sizes can be inferred correctly for all dimensions.

For the dask arrays, we have the chunking information and we can check if they can work together. As the numpy arrays are in-memory we can choose the chunking information as we like as long as they broadcast well.

What am I missing?

Nothing.  You understand the problem completely.  Now all we have to do is implement that logic in all dask.array functions.  This might be non-trivial.

I think a good first step that broadcasts a numpy array into dask.array. We could build this incrementally too. I was thinking something like

``` python
def broadcast_to_darray(np_arr, dask_arr):  # maybe we just need dask_arr.shape
    # simple scalar case
    if np.isscalarlike(np_arr):
        return da.fill(np_arr, dask_arr.shape), dask_arr
    # other cases where dask_arr.shape does not change
    # stuff where both shapes change
    raise NotImplementedError
```

Then we would stick this in `dask.array.core.elemwise`.

I had some auto chunking ideas that I wrote down in another issue. I agree though that when the shape of the dask array does not change we should use the pre-existing chunks b

How would a dask.array be chunked in a dimension that is extended? For quick reference here are the numpy broadcasting docs http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html

edit: oh @shoyer has an answer :+1: 

See https://github.com/ContinuumIO/dask/issues/111

I have a PR on this that I'm putting together -- still needs a few fixes for the tests. 

Glad to hear it :)


Resolving patch:
diff --git a/dask/array/core.py b/dask/array/core.py
--- a/dask/array/core.py
+++ b/dask/array/core.py
@@ -182,7 +182,8 @@ def zero_broadcast_dimensions(lol, nblocks):
     return deepmap(f, lol)
 
 
-def broadcast_dimensions(argpairs, numblocks, sentinels=(1, (1,))):
+def broadcast_dimensions(argpairs, numblocks, sentinels=(1, (1,)),
+                         consolidate=None):
     """ Find block dimensions from arguments
 
     Parameters
@@ -194,6 +195,8 @@ def broadcast_dimensions(argpairs, numblocks, sentinels=(1, (1,))):
         maps {name: number of blocks}
     sentinels: iterable (optional)
         values for singleton dimensions
+    consolidate: func (optional)
+        use this to reduce each set of common blocks into a smaller set
 
     Examples
     --------
@@ -221,11 +224,15 @@ def broadcast_dimensions(argpairs, numblocks, sentinels=(1, (1,))):
     L = concat([zip(inds, dims)
                     for (x, inds), (x, dims)
                     in join(first, argpairs, first, numblocks.items())])
+
     g = groupby(0, L)
     g = dict((k, set([d for i, d in v])) for k, v in g.items())
 
     g2 = dict((k, v - set(sentinels) if len(v) > 1 else v) for k, v in g.items())
 
+    if consolidate is not None:
+        g2 = valmap(consolidate, g2)
+
     if g2 and not set(map(len, g2.values())) == set([1]):
         raise ValueError("Shapes do not align %s" % g)
 
@@ -676,6 +683,8 @@ def _keys(self, *args):
             return [self._keys(*(args + (i,)))
                         for i in range(self.numblocks[ind])]
 
+    __array_priority__ = 11  # higher than numpy.ndarray and numpy.matrix
+
     def __array__(self, dtype=None, **kwargs):
         x = self.compute()
         if dtype and x.dtype != dtype:
@@ -786,6 +795,8 @@ def __complex__(self):
         return complex(self.compute())
 
     def __getitem__(self, index):
+        out = 'getitem-' + tokenize(self, index)
+
         # Field access, e.g. x['a'] or x[['a', 'b']]
         if (isinstance(index, (str, unicode)) or
             (    isinstance(index, list)
@@ -796,7 +807,7 @@ def __getitem__(self, index):
                 dt = np.dtype([(name, self._dtype[name]) for name in index])
             else:
                 dt = None
-            return elemwise(getarray, self, index, dtype=dt)
+            return elemwise(getarray, self, index, dtype=dt, name=out)
 
         # Slicing
         if not isinstance(index, tuple):
@@ -805,7 +816,6 @@ def __getitem__(self, index):
         if all(isinstance(i, slice) and i == slice(None) for i in index):
             return self
 
-        out = 'getitem-' + tokenize(self, index)
         dsk, chunks = slice_array(out, self.name, self.chunks, index)
 
         return Array(merge(self.dask, dsk), out, chunks, dtype=self._dtype)
@@ -1184,6 +1194,67 @@ def from_func(func, shape, dtype=None, name=None, args=(), kwargs={}):
     return Array(dsk, name, chunks, dtype)
 
 
+def common_blockdim(blockdims):
+    """ Find the common block dimensions from the list of block dimensions
+
+    Currently only implements the simplest possible heuristic: the common
+    block-dimension is the only one that does not span fully span a dimension.
+    This is a conservative choice that allows us to avoid potentially very
+    expensive rechunking.
+
+    Assumes that each element of the input block dimensions has all the same
+    sum (i.e., that they correspond to dimensions of the same size).
+
+    Examples
+    --------
+
+    >>> common_blockdim([(3,), (2, 1)])
+    set([(2, 1)])
+    >>> common_blockdim([(2, 2), (3, 1)])  # doctest: +SKIP
+    Traceback (most recent call last):
+        ...
+    ValueError: Chunks do not align
+    """
+    non_trivial_dims = set([d for d in blockdims if len(d) > 1])
+    if len(non_trivial_dims) > 1:
+        raise ValueError('Chunks do not align %s' % non_trivial_dims)
+    elif non_trivial_dims:
+        return non_trivial_dims
+    else:
+        return blockdims
+
+
+def unify_chunks(*args):
+    """ Unify chunks across a sequence of arrays
+
+    Currently only uses very simple rules for unifying chunks: see
+    common_blockdim for more details.
+
+    Parameters
+    ----------
+    *args: sequence of Array, index pairs
+        Sequence like (x, 'ij', y, 'jk', z, 'i')
+
+    Returns
+    -------
+    chunkss : dict
+        Map like {index: chunks}.
+    arrays : list
+        List of rechunked arrays.
+    """
+    arginds = list(partition(2, args)) # [x, ij, y, jk] -> [(x, ij), (y, jk)]
+
+    nameinds = [(a.name, i) for a, i in arginds]
+    blockdim_dict = dict((a.name, a.chunks) for a, _ in arginds)
+
+    chunkss = broadcast_dimensions(nameinds, blockdim_dict,
+                                   consolidate=common_blockdim)
+    arrays = [a.rechunk(tuple(chunkss[j] if a.shape[n] > 1 else 1
+                              for n, j in enumerate(i)))
+              for a, i in arginds]
+    return chunkss, arrays
+
+
 def atop(func, out_ind, *args, **kwargs):
     """ Tensor operation: Generalized inner and outer products
 
@@ -1256,26 +1327,20 @@ def atop(func, out_ind, *args, **kwargs):
     if kwargs:
         raise TypeError("%s does not take the following keyword arguments %s" %
             (func.__name__, str(sorted(kwargs.keys()))))
-    arginds = list(partition(2, args)) # [x, ij, y, jk] -> [(x, ij), (y, jk)]
-    numblocks = dict([(a.name, a.numblocks) for a, ind in arginds])
+
+    chunkss, arrays = unify_chunks(*args)
+    arginds = list(zip(arrays, args[1::2]))
+
+    numblocks = dict([(a.name, a.numblocks) for a, _ in arginds])
     argindsstr = list(concat([(a.name, ind) for a, ind in arginds]))
     # Finish up the name
     if not out:
         out = 'atop-' + tokenize(func, out_ind, argindsstr, dtype)
 
     dsk = top(func, out, out_ind, *argindsstr, numblocks=numblocks)
-
-    # Dictionary mapping {i: 3, j: 4, ...} for i, j, ... the dimensions
-    shapes = dict((a.name, a.shape) for a, _ in arginds)
-    nameinds = [(a.name, i) for a, i in arginds]
-    dims = broadcast_dimensions(nameinds, shapes)
-    shape = tuple(dims[i] for i in out_ind)
-
-    blockdim_dict = dict((a.name, a.chunks) for a, _ in arginds)
-    chunkss = broadcast_dimensions(nameinds, blockdim_dict)
+    dsks = [a.dask for a, _ in arginds]
     chunks = tuple(chunkss[i] for i in out_ind)
 
-    dsks = [a.dask for a, _ in arginds]
     return Array(merge(dsk, *dsks), out, chunks, dtype=dtype)
 
 
@@ -1541,6 +1606,22 @@ def store(x, index):
     return dsk
 
 
+def asarray(array):
+    """Coerce argument into a dask array
+
+    >>> x = np.arange(3)
+    >>> asarray(x)
+    dask.array<asarray-..., shape=(3,), chunks=((3,)), dtype=int64>
+    >>>
+    """
+    if not isinstance(array, Array):
+        name = 'asarray-' + tokenize(array)
+        if not hasattr(array, 'shape'):
+            array = np.asarray(array)
+        array = from_array(array, chunks=array.shape, name=name)
+    return array
+
+
 def partial_by_order(op, other):
     """
 
@@ -1566,6 +1647,28 @@ def f(*args):
     return f
 
 
+def is_scalar_for_elemwise(arg):
+    """
+    >>> is_scalar_for_elemwise(42)
+    True
+    >>> is_scalar_for_elemwise('foo')
+    True
+    >>> is_scalar_for_elemwise(True)
+    True
+    >>> is_scalar_for_elemwise(np.array(42))
+    True
+    >>> is_scalar_for_elemwise([1, 2, 3])
+    True
+    >>> is_scalar_for_elemwise(np.array([1, 2, 3]))
+    False
+    >>> is_scalar_for_elemwise(from_array(np.array(0), chunks=()))
+    False
+    """
+    return (np.isscalar(arg)
+            or not hasattr(arg, 'shape')
+            or (isinstance(arg, np.ndarray) and arg.ndim == 0))
+
+
 def elemwise(op, *args, **kwargs):
     """ Apply elementwise function across arguments
 
@@ -1583,23 +1686,26 @@ def elemwise(op, *args, **kwargs):
     if not set(['name', 'dtype']).issuperset(kwargs):
         raise TypeError("%s does not take the following keyword arguments %s" %
             (op.__name__, str(sorted(set(kwargs) - set(['name', 'dtype'])))))
-    out_ndim = max(len(arg.shape) if isinstance(arg, Array) else 0
-                   for arg in args)
+
+    out_ndim = max(len(getattr(arg, 'shape', ())) for arg in args)
     expr_inds = tuple(range(out_ndim))[::-1]
 
-    arrays = [arg for arg in args if isinstance(arg, Array)]
-    other = [(i, a) for i, a in enumerate(args) if not isinstance(a, Array)]
-    if any(isinstance(arg, np.ndarray) for arg in args):
-        raise NotImplementedError("Dask.array operations only work on dask "
-                                  "arrays, not numpy arrays.")
+    arrays = [asarray(a) for a in args if not is_scalar_for_elemwise(a)]
+    other = [(i, a) for i, a in enumerate(args) if is_scalar_for_elemwise(a)]
+
     if 'dtype' in kwargs:
         dt = kwargs['dtype']
-    elif not all(a._dtype is not None for a in arrays):
+    elif any(a._dtype is None for a in arrays):
         dt = None
     else:
-
+        # We follow NumPy's rules for dtype promotion, which special cases
+        # scalars and 0d ndarrays (which it considers equivalent) by using
+        # their values to compute the result dtype:
+        # https://github.com/numpy/numpy/issues/6240
+        # We don't inspect the values of 0d dask arrays, because these could
+        # hold potentially very expensive calculations.
         vals = [np.empty((1,) * a.ndim, dtype=a.dtype)
-                if hasattr(a, 'dtype') else a
+                if not is_scalar_for_elemwise(a) else a
                 for a in args]
         try:
             dt = op(*vals).dtype
diff --git a/dask/array/rechunk.py b/dask/array/rechunk.py
--- a/dask/array/rechunk.py
+++ b/dask/array/rechunk.py
@@ -198,6 +198,8 @@ def rechunk(x, chunks):
         chunks = tuple(lc if lc is not None else rc
                        for lc, rc in zip(chunks, x.chunks))
     chunks = normalize_chunks(chunks, x.shape)
+    if chunks == x.chunks:
+        return x
     if not len(chunks) == x.ndim or tuple(map(sum, chunks)) != x.shape:
         raise ValueError("Provided chunks are not consistent with shape")
 


Test patch:
diff --git a/dask/array/tests/test_array_core.py b/dask/array/tests/test_array_core.py
--- a/dask/array/tests/test_array_core.py
+++ b/dask/array/tests/test_array_core.py
@@ -320,10 +320,10 @@ def test_take():
 
 
 def test_binops():
-    a = Array(dict((('a', i), '') for i in range(3)),
-              'a', chunks=((10, 10, 10),))
-    b = Array(dict((('b', i), '') for i in range(3)),
-              'b', chunks=((10, 10, 10),))
+    a = Array(dict((('a', i), np.array([''])) for i in range(3)),
+              'a', chunks=((1, 1, 1),))
+    b = Array(dict((('b', i), np.array([''])) for i in range(3)),
+              'b', chunks=((1, 1, 1),))
 
     result = elemwise(add, a, b, name='c')
     assert result.dask == merge(a.dask, b.dask,
@@ -377,6 +377,32 @@ def test_partial_by_order():
     assert raises(ValueError, lambda: partial_by_order(add, 1))
     assert raises(ValueError, lambda: partial_by_order(add, [1]))
 
+
+def test_elemwise_with_ndarrays():
+    x = np.arange(3)
+    y = np.arange(12).reshape(4, 3)
+    a = from_array(x, chunks=(3,))
+    b = from_array(y, chunks=(2, 3))
+
+    assert eq(x + a, 2 * x)
+    assert eq(a + x, 2 * x)
+
+    assert eq(x + b, x + y)
+    assert eq(b + x, x + y)
+    assert eq(a + y, x + y)
+    assert eq(y + a, x + y)
+
+
+def test_elemwise_differently_chunked():
+    x = np.arange(3)
+    y = np.arange(12).reshape(4, 3)
+    a = from_array(x, chunks=(3,))
+    b = from_array(y, chunks=(2, 2))
+
+    assert eq(a + b, x + y)
+    assert eq(b + a, x + y)
+
+
 def test_operators():
     x = np.arange(10)
     y = np.arange(10).reshape((10, 1))
@@ -399,6 +425,16 @@ def test_operators():
     assert eq(a, +x)
 
 
+def test_operator_dtype_promotion():
+    x = np.arange(10, dtype=np.float32)
+    y = np.array([1])
+    a = from_array(x, chunks=(5,))
+
+    assert eq(x + 1, a + 1)  # still float32
+    assert eq(x + 1e50, a + 1e50)  # now float64
+    assert eq(x + y, a + y)  # also float64
+
+
 def test_field_access():
     x = np.array([(1, 1.0), (2, 2.0)], dtype=[('a', 'i4'), ('b', 'f4')])
     y = from_array(x, chunks=(1,))
@@ -1187,12 +1223,6 @@ def test_from_array_with_missing_chunks():
     assert d.chunks == da.from_array(x, chunks=(2, 2, 3)).chunks
 
 
-def test_numpy_compat_is_notimplemented():
-    a = np.arange(10)
-    x = da.from_array(a, chunks=5)
-    assert raises(NotImplementedError, lambda: x + a)
-
-
 def test_cache():
     x = da.arange(15, chunks=5)
     y = 2 * x + 1
diff --git a/dask/array/tests/test_rechunk.py b/dask/array/tests/test_rechunk.py
--- a/dask/array/tests/test_rechunk.py
+++ b/dask/array/tests/test_rechunk.py
@@ -177,3 +177,17 @@ def test_rechunk_with_integer():
     y = x.rechunk(3)
     assert y.chunks == ((3, 2),)
     assert (x.compute() == y.compute()).all()
+
+
+def test_rechunk_0d():
+    a = np.array(42)
+    x = da.from_array(a, chunks=())
+    y = x.rechunk(())
+    assert y.chunks == ()
+    assert y.compute() == a
+
+
+def test_rechunk_same():
+    x = da.ones((24, 24), chunks=(4, 8))
+    y = x.rechunk(x.chunks)
+    assert x is y