Repository: dask/dask

Issue:
slices of memmaps don't concatenate correctly
I have several slices of the same underlying numpy.memmap object. I want to concatenate them using dask. As shown in the example below, it doesn't work.

``` python
import dask.array as da
import numpy as np

# test data
dtype = 'f8'
data = np.arange(10, dtype=dtype)
# save as memmap
data.tofile('data.bin')
# load as memmap
mm = np.memmap('data.bin', dtype)
# slices of the memmap
mm_slices = [mm[:5], mm[5:]]
# make dask arrays of those slices
da_slices = [da.from_array(mm_sl, chunks=(5,)) for mm_sl in mm_slices]
# concatenate with dask
mm_da = da.concatenate(da_slices)
np.testing.assert_allclose(data, mm_da.compute())
```

This fails with

```
(mismatch 50.0%)
 x: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
 y: array([5, 6, 7, 8, 9, 5, 6, 7, 8, 9])
```

The first chunk has been replaced by the second chunk's data. Note this does not happen if I look at the slices individually (e.g. `da_slices[0].compute()` gives the correct values). 

I strongly suspect this is related to #1562 and the way that memmaps are tokenized. In fact, these two slices get the same token:

``` python
from dask.base import tokenize
print([tokenize(mm_sl) for mm_sl in mm_slices])
>>> ['0053a0dda7a12ecf8ad86dbf62ba1c7c', '0053a0dda7a12ecf8ad86dbf62ba1c7c']
```

This is because the `normalize_array` function checks objects for the `.filename` attribute and uses this to determine the token.

``` python
        if hasattr(x, 'mode') and hasattr(x, 'filename'):
            return x.filename, os.path.getmtime(x.filename), x.dtype, x.shape
```

Both these memmaps have the same filename, despite representing different slices, so they get the same token.

Note that the whole point of this is to _avoid reading the data into memory_. In my real application, `data.bin` would be an enormous binary file which I can't fit into memory. So any fix which has to read the whole array in order to tokenize is useless to me. At that point, I might as well use `numpy.concatenate`.

I'm not sure what the solution is here... Maybe I need to bypass `concatenate` and implement a custom function to wrap my data using the lower level constructor for `dask.array.Array`. But regardless, the tokenizing of memmap is still buggy and could cause problems for other users.



Discussion:
Hmmm, that is a problem. I'm not sure if we can fix this with deterministic tokenization, as `mmap` doesn't seem to expose any information about the underlying buffer.

For now, why don't you let `dask` handle the slicing for you? This works, and has the hashes work out properly:

``` python
In [1]: %load temp.py

In [2]: # %load temp.py
   ...: import dask.array as da
   ...: import numpy as np
   ...:
   ...: # test data
   ...: dtype = 'f8'
   ...: data = np.arange(10, dtype=dtype)
   ...: # save as memmap
   ...: data.tofile('data.bin')
   ...: # load as memmap
   ...: mm = np.memmap('data.bin', dtype)
   ...: dmm = da.from_array(mm, chunks=5)
   ...: # make dask arrays of those slices
   ...: da_slices = [dmm[:5], dmm[5:]]
   ...: # concatenate with dask
   ...: mm_da = da.concatenate(da_slices)
   ...: np.testing.assert_allclose(data, mm_da.compute())
```


Resolving patch:
diff --git a/dask/base.py b/dask/base.py
--- a/dask/base.py
+++ b/dask/base.py
@@ -316,7 +316,13 @@ def normalize_array(x):
         if not x.shape:
             return (str(x), x.dtype)
         if hasattr(x, 'mode') and getattr(x, 'filename', None):
-            return x.filename, os.path.getmtime(x.filename), x.dtype, x.shape
+            if hasattr(x.base, 'ctypes'):
+                offset = (x.ctypes.get_as_parameter().value -
+                          x.base.ctypes.get_as_parameter().value)
+            else:
+                offset = 0  # root memmap's have mmap object as base
+            return (x.filename, os.path.getmtime(x.filename), x.dtype,
+                    x.shape, x.strides, offset)
         if x.dtype.hasobject:
             try:
                 data = md5('-'.join(x.flat).encode('utf-8')).hexdigest()


Test patch:
diff --git a/dask/tests/test_base.py b/dask/tests/test_base.py
--- a/dask/tests/test_base.py
+++ b/dask/tests/test_base.py
@@ -123,6 +123,19 @@ def test_tokenize_numpy_memmap():
 
     assert y != z
 
+    with tmpfile('.npy') as fn:
+        x = np.random.normal(size=(10, 10))
+        np.save(fn, x)
+        mm = np.load(fn, mmap_mode='r')
+        mm2 = np.load(fn, mmap_mode='r')
+        a = tokenize(mm[0, :])
+        b = tokenize(mm[1, :])
+        c = tokenize(mm[0:3, :])
+        d = tokenize(mm[:, 0])
+        assert len(set([a, b, c, d])) == 4
+        assert tokenize(mm) == tokenize(mm2)
+        assert tokenize(mm[1, :]) == tokenize(mm2[1, :])
+
 
 @pytest.mark.skipif('not np')
 def test_tokenize_numpy_memmap_no_filename():