Repository: dask/dask

Issue:
ValueError: Dimension 0 has 1 blocks, chunks specified with 3 blocks
I'm getting this error when using dask 0.13.0 in PY36 with code that previously worked fine at least with dask 0.11 in PY35. It arises within a call to `da.map_blocks`, I have provided a `chunks` kwarg and I know it is correct and matches with the input arrays, but I think there is a bug in computation of the `numblocks` variable within `da.map_blocks`. I think the problem is in [this line of code](https://github.com/dask/dask/blob/master/dask/array/core.py#L642), I think `numblocks` ends up in the wrong order, although I don't understand why. I'll try to work up a minimal example but thought I'd post an issue now in case it's something fairly obvious.


Discussion:
Nothing in particular jumps out at me.  Map blocks has become a bit messier as people have asked for more and more features on it, so it's not terribly surprising to see corner cases develop.  My apologies for the development dust.
No worries, here's an example:

```
Python 3.6.0 | packaged by conda-forge | (default, Jan 13 2017, 23:17:12) 
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import dask
>>> dask.__version__
'0.13.0'
>>> import dask.array as da
>>> import numpy as np
>>> np.__version__
'1.11.3'
>>> x = np.ones((5, 3))
>>> y = np.ones((5, 3))
>>> dx = da.from_array(x, chunks=(2, 2))
>>> dy = da.from_array(y, chunks=(2, 2))
>>> dz = da.map_blocks(np.add, dx, dy)
>>> dz
dask.array<add-d3f..., shape=(5, 3), dtype=float64, chunksize=(2, 2)>
>>> dz.compute()
array([[ 2.,  2.,  2.],
       [ 2.,  2.,  2.],
       [ 2.,  2.,  2.],
       [ 2.,  2.,  2.],
       [ 2.,  2.,  2.]])
>>> dz = da.map_blocks(np.add, dx, dy, chunks=dx.chunks)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/aliman/miniconda3/envs/dev-scikit-allel-py36/lib/python3.6/site-packages/dask/array/core.py", line 665, in map_blocks
    "{2} blocks".format(i, nb, len(c)))
ValueError: Dimension 0 has 2 blocks, chunks specified with 3 blocks
```

It works when I don't provide `chunks` to `map_blocks`, but get an error when I do provide `chunks`. In my actual use case, I tried leaving out `chunks` and letting dask infer, but dask tells me it cannot infer and so `chunks` must be provided.

The error reports dimension 0 has 2 blocks but clearly it has three, and dimension 1 has 2 blocks, hence I suspect the `numblocks` variable within `map_blocks` ends up in the wrong order in this case.
Odd, this test passes for me:

```python
def test_map_blocks_chunkshape():
    dx = da.ones((5, 3), chunks=(2, 2))
    dy = da.ones((5, 3), chunks=(2, 2))
    dz = da.map_blocks(np.add, dx, dy, chunks=dx.chunks)
    assert_eq(dz, np.ones((5, 3)) * 2)
```

Installing different versions of things now.
cc @jcrist 
Hmmm, I'm also unable to reproduce:

```python
In [1]: import numpy as np

In [2]: np.__version__
Out[2]: '1.11.3'

In [3]: import dask

In [4]: dask.__version__
Out[4]: u'0.13.0'

In [5]: import dask.array as da

In [6]: x = np.ones((5, 3))

In [7]: y = np.ones((5, 3))

In [8]: dx = da.from_array(x, chunks=(2, 2))

In [9]: dy = da.from_array(x, chunks=(2, 2))

In [10]: dz = da.map_blocks(np.add, dx, dy, chunks=dx.chunks)

In [11]: dz.compute()
Out[11]:
array([[ 2.,  2.,  2.],
       [ 2.,  2.,  2.],
       [ 2.,  2.,  2.],
       [ 2.,  2.,  2.],
       [ 2.,  2.,  2.]])
```
Nevermind, I can only reproduce in python 3.6 (not in 3.5). Looking now.
I believe implementation of dict has changed in PY36, maybe ordering of
items is different?

On Tuesday, January 31, 2017, Jim Crist <notifications@github.com> wrote:

> Nevermind, I can only reproduce in python 3.6 (not in 3.5). Looking now.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/dask/issues/1947#issuecomment-276455042>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AAq8QvHDNvBZSz_SIUkWJ_F2gQNBcjtOks5rX4KWgaJpZM4LyAwV>
> .
>


-- 
Alistair Miles
Head of Epidemiological Informatics
Centre for Genomics and Global Health <http://cggh.org>
The Wellcome Trust Centre for Human Genetics
Roosevelt Drive
Oxford
OX3 7BN
United Kingdom
Email: alimanfoo@googlemail.com
Web: http://purl.org/net/aliman
Twitter: https://twitter.com/alimanfoo
Tel: +44 (0)1865 287721

Yeah, I figured it out. Things just happened to work in our examples in the past due to luck - we were calling `reversed` on dictionary items, when we meant `sorted(dict.items(), reverse=True)`. PR momentarily.

Resolving patch:
diff --git a/dask/array/core.py b/dask/array/core.py
--- a/dask/array/core.py
+++ b/dask/array/core.py
@@ -639,7 +639,7 @@ def map_blocks(func, *args, **kwargs):
         numblocks = list(arrs[0].numblocks)
     else:
         dims = broadcast_dimensions(argpairs, numblocks)
-        numblocks = [b for (_, b) in reversed(list(dims.items()))]
+        numblocks = [b for (_, b) in sorted(dims.items(), reverse=True)]
 
     if drop_axis:
         if any(numblocks[i] > 1 for i in drop_axis):


Test patch:
diff --git a/dask/array/tests/test_array_core.py b/dask/array/tests/test_array_core.py
--- a/dask/array/tests/test_array_core.py
+++ b/dask/array/tests/test_array_core.py
@@ -845,6 +845,13 @@ def test_map_blocks_with_kwargs():
               np.array([4, 9]))
 
 
+def test_map_blocks_with_chunks():
+    dx = da.ones((5, 3), chunks=(2, 2))
+    dy = da.ones((5, 3), chunks=(2, 2))
+    dz = da.map_blocks(np.add, dx, dy, chunks=dx.chunks)
+    assert_eq(dz, np.ones((5, 3)) * 2)
+
+
 def test_map_blocks_dtype_inference():
     x = np.arange(50).reshape((5, 10))
     y = np.arange(10)