Repository: dask/dask

Issue:
stack/concatenate duplicates last result in list comprehension
Am running into some strange behavior with Dask `stack` and `concatenate` where they end up duplicating the last value in a list comprehension for all other values in the stacked or concatenated array. Not seeing this across the board, but am seeing it reliably in a few cases. Not sure if there is some optimization gone awry or something else giving rise to this problem. Have documented it in issue ( https://github.com/dask-image/dask-ndfilters/issues/28 ) with an example and specs for reproducing the issue.


Discussion:
Can you provide a reproducible example without the `dask-ndfilters` package?
I'll definitely look into it. However simple `stack` and `concatenate` operations on Dask arrays didn't seem to be cutting it.

That said, the fact that the comprehensions using `dask-ndfilters` behave correctly with `numpy.stack` and `numpy.concatenate`, but not with `da.stack` and `da.concatenate` seems to suggest there is something here that needs investigating.

Considering I'm a bit stumped on this one, figured two heads would be better than one. 😄
Alright I think I have found a reproducer with Dask alone. It seems that specifying the `name` argument to [`map_blocks`]( http://dask.pydata.org/en/latest/array-api.html#dask.array.Array.map_blocks ) is messing things up. In `dask-ndfilters`, the functions provide their names for `name` to make it easy to find and debug them in the graph.

Didn't realize we had to do our own hashing and tack it on to this name as well. Would it be possible for Dask to do this hashing for us when the `name` is set?

<br>

Code snippet:

<details>

```python
In [1]: import numpy as np

In [2]: import dask.array as da

In [3]: a = np.random.random((10, 12, 14))

In [4]: d = da.from_array(a, chunks=(5, 6, 7))

In [5]: d1 = da.stack([d[i].map_blocks(lambda x: x) for i in range(len(d))])

In [6]: d2 = da.stack([d[i].map_blocks(lambda x: x, name="ident") for i in range(len(d))])

In [7]: (d.compute() == d1.compute()).all()
Out[7]: True

In [8]: (d.compute() == d2.compute()).all()
Out[8]: False

In [9]: (d[-1].compute() == d2[-1].compute()).all()
Out[9]: True

In [10]: (d[-1].compute() == d2[0].compute()).all()
Out[10]: True
```

</details>

<br>
Environment:
<br>
<br>

<details>

```yaml
name: daskenv
channels: !!python/tuple
- !!python/unicode
  'conda-forge'
- !!python/unicode
  'defaults'
dependencies:
- conda-forge::alabaster=0.7.10=py27_0
- conda-forge::appnope=0.1.0=py27_0
- conda-forge::babel=2.3.4=py27_0
- conda-forge::backports.lzma=0.0.8=py27_0
- conda-forge::backports.shutil_get_terminal_size=1.0.0=py27_1
- conda-forge::backports_abc=0.5=py27_0
- conda-forge::blas=1.1=openblas
- conda-forge::bokeh=0.12.5=py27_2
- conda-forge::boto3=1.4.4=py27_0
- conda-forge::botocore=1.5.26=py27_0
- conda-forge::ca-certificates=2017.1.23=1
- conda-forge::certifi=2017.1.23=py27_0
- conda-forge::click=6.7=py27_0
- conda-forge::cloudpickle=0.2.2=py27_2
- conda-forge::configparser=3.5.0=py27_0
- conda-forge::coverage=4.3.4=py27_0
- conda-forge::cycler=0.10.0=py27_0
- conda-forge::dask=0.14.3=py27_0
- conda-forge::decorator=4.0.11=py27_0
- conda-forge::distributed=1.16.0=py27_0
- conda-forge::docutils=0.13.1=py27_0
- conda-forge::enum34=1.1.6=py27_1
- conda-forge::flake8=3.3.0=py27_0
- conda-forge::freetype=2.7=1
- conda-forge::functools32=3.2.3.2=py27_1
- conda-forge::futures=3.0.5=py27_0
- conda-forge::graphviz=2.38.0=4
- conda-forge::h5py=2.6.0=np112py27_7
- conda-forge::hdf5=1.8.17=9
- conda-forge::heapdict=1.0.0=py27_0
- conda-forge::imagesize=0.7.1=py27_0
- conda-forge::ipython=5.3.0=py27_0
- conda-forge::ipython_genutils=0.2.0=py27_0
- conda-forge::jinja2=2.9.5=py27_0
- conda-forge::jmespath=0.9.2=py27_0
- conda-forge::jpeg=9b=0
- conda-forge::libgfortran=3.0.0=0
- conda-forge::libpng=1.6.28=0
- conda-forge::libtiff=4.0.6=7
- conda-forge::locket=0.2.0=py27_1
- conda-forge::markupsafe=0.23=py27_1
- conda-forge::matplotlib=2.0.1=np112py27_0
- conda-forge::mccabe=0.6.1=py27_0
- conda-forge::msgpack-python=0.4.8=py27_0
- conda-forge::ncurses=5.9=10
- conda-forge::numpy=1.12.1=py27_blas_openblas_200
- conda-forge::numpydoc=0.6.0=py27_0
- conda-forge::openblas=0.2.19=1
- conda-forge::openssl=1.0.2k=0
- conda-forge::pandas=0.19.2=np112py27_1
- conda-forge::partd=0.3.7=py27_0
- conda-forge::pathlib2=2.2.1=py27_0
- conda-forge::pexpect=4.2.1=py27_0
- conda-forge::pickleshare=0.7.3=py27_0
- conda-forge::pip=9.0.1=py27_0
- conda-forge::prompt_toolkit=1.0.13=py27_0
- conda-forge::psutil=5.2.0=py27_0
- conda-forge::ptyprocess=0.5.1=py27_0
- conda-forge::py=1.4.33=py27_0
- conda-forge::pycodestyle=2.3.1=py27_0
- conda-forge::pyfftw=0.10.4=np112py27_1
- conda-forge::pyflakes=1.5.0=py27_0
- conda-forge::pygments=2.2.0=py27_0
- conda-forge::pyparsing=2.2.0=py27_0
- conda-forge::pytest=3.0.7=py27_0
- conda-forge::pytest-cov=2.4.0=py27_0
- conda-forge::python=2.7.13=0
- conda-forge::python-dateutil=2.6.0=py27_0
- conda-forge::python-graphviz=0.6=py27_1
- conda-forge::pytz=2016.10=py27_0
- conda-forge::pyyaml=3.12=py27_0
- conda-forge::readline=6.2=0
- conda-forge::requests=2.13.0=py27_0
- conda-forge::s3fs=0.0.9=py27_0
- conda-forge::s3transfer=0.1.10=py27_0
- conda-forge::scandir=1.5=py27_1
- conda-forge::scipy=0.19.0=np112py27_blas_openblas_200
- conda-forge::setuptools=33.1.1=py27_0
- conda-forge::simplegeneric=0.8.1=py27_0
- conda-forge::singledispatch=3.4.0.3=py27_0
- conda-forge::six=1.10.0=py27_1
- conda-forge::snowballstemmer=1.2.1=py27_0
- conda-forge::sortedcontainers=1.5.3=py27_0
- conda-forge::sphinx=1.5.4=py27_0
- conda-forge::sphinx_rtd_theme=0.2.4=py27_0
- conda-forge::sqlite=3.13.0=1
- conda-forge::ssl_match_hostname=3.5.0.1=py27_1
- conda-forge::subprocess32=3.2.7=py27_0
- conda-forge::tblib=1.3.2=py27_0
- conda-forge::tk=8.5.19=1
- conda-forge::toolz=0.8.2=py27_0
- conda-forge::tornado=4.4.2=py27_0
- conda-forge::traitlets=4.3.2=py27_0
- conda-forge::wcwidth=0.1.7=py27_0
- conda-forge::wheel=0.29.0=py27_0
- conda-forge::xz=5.2.2=0
- conda-forge::yaml=0.1.6=0
- conda-forge::zict=0.1.1=py27_0
- conda-forge::zlib=1.2.11=0
- pip:
  - backports-abc==0.5
  - backports.shutil-get-terminal-size==1.0.0
  - backports.ssl-match-hostname==3.5.0.1
  - ipython-genutils==0.2.0
  - prompt-toolkit==1.0.13
  - sphinx-rtd-theme==0.2.4
```

</details>
Ah, yes, if you provide a `name` you take on the responsibility of hashing the arguments and need to make sure it's unique. I can see the desire for only specifying the prefix though. I'll add a keyword (`da.atop` uses `token`) for that and clarify the docs.
Thanks @jcrist. That proposal sounds good. Sorry for overlooking this in the docs initially.

Resolving patch:
diff --git a/dask/array/core.py b/dask/array/core.py
--- a/dask/array/core.py
+++ b/dask/array/core.py
@@ -533,9 +533,13 @@ def map_blocks(func, *args, **kwargs):
     new_axis : number or iterable, optional
         New dimensions created by the function. Note that these are applied
         after ``drop_axis`` (if present).
+    token : string, optional
+        The key prefix to use for the output array. If not provided, will be
+        determined from the function name.
     name : string, optional
-        The key name to use for the array. If not provided, will be determined
-        by a hash of the arguments.
+        The key name to use for the output array. Note that this fully
+        specifies the output key name, and must be unique. If not provided,
+        will be determined by a hash of the arguments.
     **kwargs :
         Other keyword arguments to pass to function. Values must be constants
         (not dask.arrays)
@@ -608,10 +612,11 @@ def map_blocks(func, *args, **kwargs):
     >>> def func(block, block_id=None):
     ...     pass
 
-    You may specify the name of the resulting task in the graph with the
-    optional ``name`` keyword argument.
+    You may specify the key name prefix of the resulting task in the graph with
+    the optional ``token`` keyword argument.
 
-    >>> y = x.map_blocks(lambda x: x + 1, name='increment')
+    >>> x.map_blocks(lambda x: x + 1, token='increment')  # doctest: +SKIP
+    dask.array<increment, shape=(100,), dtype=int64, chunksize=(10,)>
     """
     if not callable(func):
         msg = ("First argument must be callable function, not %s\n"
@@ -619,7 +624,10 @@ def map_blocks(func, *args, **kwargs):
                "   or:   da.map_blocks(function, x, y, z)")
         raise TypeError(msg % type(func).__name__)
     name = kwargs.pop('name', None)
-    name = name or '%s-%s' % (funcname(func), tokenize(func, args, **kwargs))
+    token = kwargs.pop('token', None)
+    if not name:
+        name = '%s-%s' % (token or funcname(func),
+                          tokenize(token or func, args, **kwargs))
     dtype = kwargs.pop('dtype', None)
     chunks = kwargs.pop('chunks', None)
     drop_axis = kwargs.pop('drop_axis', [])


Test patch:
diff --git a/dask/array/tests/test_array_core.py b/dask/array/tests/test_array_core.py
--- a/dask/array/tests/test_array_core.py
+++ b/dask/array/tests/test_array_core.py
@@ -895,6 +895,10 @@ def test_map_blocks():
     e = d.map_blocks(inc, name='increment')
     assert e.name == 'increment'
 
+    e = d.map_blocks(inc, token='increment')
+    assert e.name != 'increment'
+    assert e.name.startswith('increment')
+
     d = from_array(x, chunks=(10, 10))
     e = d.map_blocks(lambda x: x[::2, ::2], chunks=(5, 5), dtype=d.dtype)