Repository: dask/dask

Issue:
non-unique names in da.from_array cause problems with da.stack
This is an obscure problem which I can't reproduce using a simple example. I thought I would open an issue anyway.

I have an experimental [module](https://github.com/xgcm/xgcm/blob/master/xgcm/mdsxray.py) that generates dask arrays from a binary file store and feeds them to xray with the proper metdata

It loops through a timeseries of files and does something roughly like this

``` python
vardata = []
# build a list of dask arrays
for n in range(timesteps):
    data = function_to_load_binary_data(n)
    myda = da.from_array(data, data.shape)
    vardata.append(myda)
# stack them
vardask = da.stack(vardata)
```

The problem is that, evidently, the dask name attribute is not unique for the different arrays in the list, leading to problems with stack. If I do

``` python
for n in range(len(vardata)):
    print 'vardata[%g] sum %g, name %s' % (n, 
        vardata[n].sum(), vardata[n].name)
    print 'vardask[%g] sum %g' % (n, 
        vardask[n].sum())
```

I get the following output

```
vardata[0] sum 22706.5, name from-array-a1d722948a08ee25f34c508fb122d765
vardask[0] sum 14416.1
vardata[1] sum 14416.1, name from-array-a1d722948a08ee25f34c508fb122d765
vardask[1] sum 14416.1
```

The two elements of the list are different, yet they got assigned the exact same name by dask. When the stack step happens, the first value is apparently overwritten by the second one, because they have the same name.

I was able to fix this in xgcm/xgcm#9 by assigning a custom `name` attribute when calling `da.from_array`. But it seems like dask should automatically be choosing a unique name.



Discussion:
Dask.array determines keynames by hashing down the contents of the array (see https://github.com/blaze/dask/blob/master/dask/base.py#L176-L189).

So if two of your input arrays have the same content (e.g. same values, shape, strides, dtype) then they will be mapped to the same key.

In principle this shouldn't affect correctness at all.  Here is a small example.

``` python
In [1]: import numpy as np
In [2]: x = np.ones((3, 3), dtype='f8')  # we'll reuse these twice
In [3]: y = np.random.random((3, 3))

In [4]: import dask.array as da
In [5]: arrays = [da.from_array(arr, chunks=arr.shape) for arr in [x, y, x, y]]
In [6]: X = da.stack(arrays, axis=0)

In [7]: X
Out[7]: dask.array<stack-b..., shape=(4, 3, 3), dtype=float64, chunksize=(1, 3, 3)>

In [8]: X.dask
Out[8]: 
{'from-array-e8122f0f6208166b3fda69f5ce877778': array([[ 1.,  1.,  1.],
        [ 1.,  1.,  1.],
        [ 1.,  1.,  1.]]),
 ('stack-bce041198f326d03aa91a3efe5d308b3',
  1,
  0,
  0): (<function dask.array.core.getarray>, ('from-array-aef6142b7e21f5c857c622429c2d672f',
   0,
   0), (None, slice(None, None, None), slice(None, None, None))),
 ('stack-bce041198f326d03aa91a3efe5d308b3',
  2,
  0,
  0): (<function dask.array.core.getarray>, ('from-array-e8122f0f6208166b3fda69f5ce877778',
   0,
   0), (None, slice(None, None, None), slice(None, None, None))),
 'from-array-aef6142b7e21f5c857c622429c2d672f': array([[ 0.66002451,  0.69792289,  0.72803412],
        [ 0.17911087,  0.74160782,  0.93489879],
        [ 0.42279052,  0.6404918 ,  0.2672002 ]]),
 ('stack-bce041198f326d03aa91a3efe5d308b3',
  3,
  0,
  0): (<function dask.array.core.getarray>, ('from-array-aef6142b7e21f5c857c622429c2d672f',
   0,
   0), (None, slice(None, None, None), slice(None, None, None))),
 ('from-array-aef6142b7e21f5c857c622429c2d672f',
  0,
  0): (<function dask.array.core.getarray>, 'from-array-aef6142b7e21f5c857c622429c2d672f', (slice(0, 3, None),
   slice(0, 3, None))),
 ('from-array-e8122f0f6208166b3fda69f5ce877778',
  0,
  0): (<function dask.array.core.getarray>, 'from-array-e8122f0f6208166b3fda69f5ce877778', (slice(0, 3, None),
   slice(0, 3, None))),
 ('stack-bce041198f326d03aa91a3efe5d308b3',
  0,
  0,
  0): (<function dask.array.core.getarray>, ('from-array-e8122f0f6208166b3fda69f5ce877778',
   0,
   0), (None, slice(None, None, None), slice(None, None, None)))}

In [9]: X.visualize(filename='X.png')
```

![x](https://cloud.githubusercontent.com/assets/306380/11126187/9d413662-8922-11e5-868f-310df0625e6d.png)

``` python
In [10]: x.sum() * 2 + y.sum() * 2
Out[10]: 28.544163038481074

In [11]: X.sum().compute()
Out[11]: 28.544163038481074
```

So anyway, that's why this _shouldn't_ be a problem.  Of course, apparently it is.  My guess is that some particular array function is failing handle repeat keys well.  Do you have any other information on when you get wrong results?

Thanks for this great explanation. It seems like the hashing is not picking up on the difference between the two arrays.

> Do you have any other information on when you get wrong results?

It is a really intermittent problem. I will try to narrow it down further.

It would be valuable to see which arrays collide.

``` python
from toolz import groupby
from dask.base import tokenize

groups = groupby(tokenize, arrays)
collisions = {k: v for k, v in groups.items() if len(v) > 1}
```

I think the trouble is relying `repr` to be uniquely identifying for custom array objets. It probably would be a better idea to default to using some sort of unique ID for each array. Here's a concrete example of what can go wrong if two arrays shares the same repr:

``` python
import numpy as np
from dask.base import tokenize

class CustomArray(object):
    def __init__(self, array):
        self.array = array
    def __getitem__(self, key):
        return self.array[key]
    @property
    def shape(self):
        return self.array.shape
    def __repr__(self):
        return repr(self.array)

# x is bigger than the amount of data numpy prints by default
x = np.zeros((1000, 1000))
y = x.copy()
y[500, 500] = 1

print tokenize(x)  # 0628ea46aa6e487c0329f05a0829d567
print tokenize(y)  # 82d43a740f5d20ac3fe4109514025bd2
print tokenize(CustomArray(x))  # 419522e29fc54c6d89e1f73bd5e2ee85
print tokenize(CustomArray(y))  # 419522e29fc54c6d89e1f73bd5e2ee85
```

Notice that the last two examples have the same token, even though they have different values!

Oh, we're not using numpy arrays then but some special class?  For ideal hashing support you would want to extend tokenize to that special class (see the example above for ndarrays).

Given that this is unlikely to happen in all cases then maybe `tokenize` should default to creating a `uuid` for safety's sake.

@rabernat is loading data from disk in some custom serialization format (using mmap?) so it's not really feasible to hash each array before putting it in dask -- that would sort of defeat the point of using dask for out of core :). I agree that defaulting to a unique ID would be a good idea for unrecognized classes.

In cases like this I usually tokenize by the filename and the modification time.  If this is a format you're likely to use a fair amount going forward then it might be worth specifying this directly.

I'll swap out the default behavior for tokenize in a bit.


Resolving patch:
diff --git a/dask/base.py b/dask/base.py
--- a/dask/base.py
+++ b/dask/base.py
@@ -1,14 +1,17 @@
 from __future__ import absolute_import, division, print_function
 
-import warnings
-from operator import attrgetter
-from hashlib import md5
 from functools import partial
+from hashlib import md5
+from operator import attrgetter
+import os
+import sys
+import uuid
+import warnings
 
-from toolz import merge, groupby, curry
+from toolz import merge, groupby, curry, identity
 from toolz.functoolz import Compose
 
-from .compatibility import bind_method
+from .compatibility import bind_method, unicode
 from .context import _globals
 from .utils import Dispatch, ignoring
 
@@ -139,10 +142,29 @@ def normalize_function(func):
 
 
 normalize_token = Dispatch()
-normalize_token.register((int, float, str, tuple, list), lambda a: a)
-normalize_token.register(object,
-        lambda a: normalize_function(a) if callable(a) else a)
-normalize_token.register(dict, lambda a: tuple(sorted(a.items())))
+normalize_token.register((int, float, str, unicode, bytes, type(None), type,
+                          slice),
+                         identity)
+
+@partial(normalize_token.register, dict)
+def normalize_dict(d):
+    return normalize_token(sorted(d.items()))
+
+@partial(normalize_token.register, (tuple, list, set))
+def normalize_seq(seq):
+    return type(seq).__name__, list(map(normalize_token, seq))
+
+@partial(normalize_token.register, object)
+def normalize_object(o):
+    if callable(o):
+        return normalize_function(o)
+    else:
+        return uuid.uuid4().hex
+
+@partial(normalize_token.register, Base)
+def normalize_base(b):
+    return type(b).__name__, b.key
+
 
 with ignoring(ImportError):
     import pandas as pd
@@ -176,6 +198,8 @@ def normalize_dataframe(df):
     def normalize_array(x):
         if not x.shape:
             return (str(x), x.dtype)
+        if hasattr(x, 'mode') and hasattr(x, 'filename'):
+            return x.filename, os.path.getmtime(x.filename), x.dtype, x.shape
         if x.dtype.hasobject:
             try:
                 data = md5('-'.join(x.flat)).hexdigest()
@@ -188,12 +212,22 @@ def normalize_array(x):
                 data = md5(x.copy().ravel().view('i1').data).hexdigest()
         return (data, x.dtype, x.shape, x.strides)
 
+    normalize_token.register(np.dtype, repr)
+    normalize_token.register(np.generic, repr)
+
+
+with ignoring(ImportError):
+    from collections import OrderedDict
+    @partial(normalize_token.register, OrderedDict)
+    def normalize_ordered_dict(d):
+        return type(d).__name__, normalize_token(list(d.items()))
+
 
 def tokenize(*args, **kwargs):
     """ Deterministic token
 
     >>> tokenize([1, 2, '3'])
-    '9d71491b50023b06fc76928e6eddb952'
+    '7d6a880cd9ec03506eee6973ff551339'
 
     >>> tokenize('Hello') == tokenize('Hello')
     True
diff --git a/dask/imperative.py b/dask/imperative.py
--- a/dask/imperative.py
+++ b/dask/imperative.py
@@ -1,9 +1,10 @@
 from __future__ import absolute_import, division, print_function
 
-import operator
+from collections import Iterator
 from functools import wraps
 from itertools import chain, count
-from collections import Iterator
+import operator
+import uuid
 
 from toolz import merge, unique, curry
 
@@ -89,9 +90,6 @@ def to_task_dasks(expr):
     return expr, []
 
 
-tokens = ('_{0}'.format(i) for i in count(1))
-
-
 def tokenize(*args, **kwargs):
     """Mapping function from task -> consistent name.
 
@@ -106,7 +104,8 @@ def tokenize(*args, **kwargs):
     """
     if kwargs.pop('pure', False):
         return base.tokenize(*args)
-    return next(tokens)
+    else:
+        return str(uuid.uuid4())
 
 
 def applyfunc(func, args, kwargs, pure=False):


Test patch:
diff --git a/dask/tests/test_base.py b/dask/tests/test_base.py
--- a/dask/tests/test_base.py
+++ b/dask/tests/test_base.py
@@ -9,14 +9,7 @@
 import dask
 from dask.base import (compute, tokenize, normalize_token, normalize_function,
         visualize)
-from dask.utils import raises
-
-
-def test_normalize():
-    assert normalize_token((1, 2, 3)) == (1, 2, 3)
-    assert normalize_token('a') == 'a'
-    assert normalize_token({'a': 1, 'b': 2, 'c': 3}) ==\
-            (('a', 1), ('b', 2), ('c', 3))
+from dask.utils import raises, tmpfile, ignoring
 
 
 def test_normalize_function():
@@ -39,9 +32,7 @@ def f3(a):
 
 def test_tokenize():
     a = (1, 2, 3)
-    b = {'a': 1, 'b': 2, 'c': 3}
-    assert tokenize(a) == '4889c6ccd7099fc2fd19f4be468fcfa0'
-    assert tokenize(a, b) == tokenize(normalize_token(a), normalize_token(b))
+    assert isinstance(tokenize(a), (str, bytes))
 
 
 def test_tokenize_numpy_array_consistent_on_values():
@@ -67,7 +58,9 @@ def test_tokenize_numpy_datetime():
 
 def test_tokenize_numpy_scalar():
     np = pytest.importorskip('numpy')
-    tokenize(np.array(1.0, dtype='f8'))
+    assert tokenize(np.array(1.0, dtype='f8')) == tokenize(np.array(1.0, dtype='f8'))
+    assert (tokenize(np.array([(1, 2)], dtype=[('a', 'i4'), ('b', 'i8')])[0])
+         == tokenize(np.array([(1, 2)], dtype=[('a', 'i4'), ('b', 'i8')])[0]))
 
 
 def test_tokenize_numpy_array_on_object_dtype():
@@ -80,6 +73,26 @@ def test_tokenize_numpy_array_on_object_dtype():
            tokenize(np.array([(1, 'a'), (1, None), (1, 'aaa')], dtype=object))
 
 
+def test_tokenize_numpy_memmap():
+    np = pytest.importorskip('numpy')
+    with tmpfile('.npy') as fn:
+        x = np.arange(5)
+        np.save(fn, x)
+        y = tokenize(np.load(fn, mmap_mode='r'))
+
+    with tmpfile('.npy') as fn:
+        x = np.arange(5)
+        np.save(fn, x)
+        z = tokenize(np.load(fn, mmap_mode='r'))
+
+    assert y != z
+
+
+def test_normalize_base():
+    for i in [1, 1.1, '1', slice(1, 2, 3)]:
+        assert normalize_token(i) is i
+
+
 def test_tokenize_pandas():
     a = pd.DataFrame({'x': [1, 2, 3], 'y': ['4', 'asd', None]}, index=[1, 2, 3])
     b = pd.DataFrame({'x': [1, 2, 3], 'y': ['4', 'asd', None]}, index=[1, 2, 3])
@@ -102,6 +115,39 @@ def test_tokenize_kwargs():
     assert tokenize(5, x=1) != tokenize(5, x=2)
     assert tokenize(5, x=1) != tokenize(5, y=1)
 
+
+def test_tokenize_same_repr():
+    class Foo(object):
+        def __init__(self, x):
+            self.x = x
+        def __repr__(self):
+            return 'a foo'
+
+    assert tokenize(Foo(1)) != tokenize(Foo(2))
+
+
+def test_tokenize_sequences():
+    assert tokenize([1]) != tokenize([2])
+    assert tokenize([1]) != tokenize((1,))
+    assert tokenize([1]) == tokenize([1])
+
+    x = np.arange(2000)  # long enough to drop information in repr
+    y = np.arange(2000)
+    y[1000] = 0  # middle isn't printed in repr
+    assert tokenize([x]) != tokenize([y])
+
+
+def test_tokenize_ordered_dict():
+    with ignoring(ImportError):
+        from collections import OrderedDict
+        a = OrderedDict([('a', 1), ('b', 2)])
+        b = OrderedDict([('a', 1), ('b', 2)])
+        c = OrderedDict([('b', 2), ('a', 1)])
+
+        assert tokenize(a) == tokenize(b)
+        assert tokenize(a) != tokenize(c)
+
+
 da = pytest.importorskip('dask.array')
 import numpy as np