Repository: dask/dask

Issue:
error tokenizing memmaps
From pydata/xarray#1014...

I have discovered that dask can't tokenize arrays that are derived from numpy memmaps (although it can tokenize the original memmaps).

Here is a minimal example:

``` python
from dask.base import tokenize
import numpy as np

# test data
shape = (10,)
dtype = np.dtype('f8')

# create some binary files
[np.zeros(shape, dtype).tofile('data%g.bin' % n) for n in range(2)]

# open them as memmaps
mm = [np.memmap('data%g.bin' % n, dtype, mode='r') for n in range(2)]

# tokenize each memmap, no problem
[tokenize(mmap) for mmap in mm]

# tokenize the products --> raises error
tokenize(mm[0] * mm[1])
```

the error I get is

``` python
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-17-df918cc6dcda> in <module>()
      9 # tokenize each memmap, no problem
     10 [tokenize(mmap) for mmap in mm]
---> 11 tokenize(mm[0] * mm[1])

/Users/rpa/anaconda/lib/python2.7/site-packages/dask/base.pyc in tokenize(*args, **kwargs)
    238     if kwargs:
    239         args = args + (kwargs,)
--> 240     return md5(str(tuple(map(normalize_token, args))).encode()).hexdigest()

/Users/rpa/anaconda/lib/python2.7/site-packages/dask/utils.pyc in __call__(self, arg)
    477         for cls in inspect.getmro(typ)[1:]:
    478             if cls in lk:
--> 479                 return lk[cls](arg)
    480         raise TypeError("No dispatch for {0} type".format(typ))
    481 

/Users/rpa/anaconda/lib/python2.7/site-packages/dask/base.pyc in normalize_array(x)
    203             return (str(x), x.dtype)
    204         if hasattr(x, 'mode') and hasattr(x, 'filename'):
--> 205             return x.filename, os.path.getmtime(x.filename), x.dtype, x.shape
    206         if x.dtype.hasobject:
    207             try:

/Users/rpa/anaconda/lib/python2.7/genericpath.pyc in getmtime(filename)
     60 def getmtime(filename):
     61     """Return the last modification time of a file, reported by os.stat()."""
---> 62     return os.stat(filename).st_mtime
     63 
     64 

TypeError: coercing to Unicode: need string or buffer, NoneType found
```



Discussion:
Thanks for the great error report with the reproducible example.

The relevant code is here:

``` python
    @partial(normalize_token.register, np.ndarray)
    def normalize_array(x):
        if not x.shape:
            return (str(x), x.dtype)
        if hasattr(x, 'mode') and hasattr(x, 'filename'):
            return x.filename, os.path.getmtime(x.filename), x.dtype, x.shape
        if x.dtype.hasobject:
            try:
                data = md5('-'.join(x.flat).encode('utf-8')).hexdigest()
            except TypeError:
                data = md5(b'-'.join([str(item).encode() for item in x.flat])). hexdigest()
        else:
            try:
                data = md5(x.ravel().view('i1').data).hexdigest()
            except (BufferError, AttributeError, ValueError):
                data = md5(x.copy().ravel().view('i1').data).hexdigest()
        return (data, x.dtype, x.shape, x.strides)
```

Question: where does `mm[0] * mm[1]` store data?  Or more specifically, what are all of the pieces of information we should hash down to produce a decent token.  Currently we identify memmap files by the presence of a `.filename` attribute.  The product of memmap files has such an attribute, but it is None.  Where is it storing its data?

> Where is it storing its data?

I wish I knew! Memmaps are weird.

I would treat the object as a regular numpy array if the `.filename` attribute is `None` (i.e. the final `else` block in ths function).

But does that mean reading the all the data in order to generate a token? Seems inefficient to me

> I would treat the object as a regular numpy array if the .filename attribute is None (i.e. the final else block in ths function).

This was my first instinct as well

> But does that mean reading the all the data in order to generate a token? Seems inefficient to me

Yes, but the data might already be in RAM in this case (assuming that memmap files don't do something special)

`numpy.memmap` is just a `numpy.ndarray` implemented in (relatively few) lines of Python:
https://github.com/numpy/numpy/blob/master/numpy/core/memmap.py

So operations like `mm[0] * mm[1]` are definitely storing the result in memory, not on disk. Given this, the fact that the result is another memmap array does seem rather strange (maybe a numpy bug).

Looks like we should treat `.filename == None` cases as normal numpy cases.

Actually, it looks like this bug has already been fixed on the dev version of numpy:

```
In [1]: import numpy as np

In [2]: mm = [np.memmap('data%g.bin' % n, float, mode='r') for n in range(2)]

In [3]: mm[0] + 1
Out[3]: array([ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.])

In [4]: mm[0]
Out[4]: memmap([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.])

In [5]: mm[0] + mm[0]
Out[5]: array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.])

In [6]: np.__version__
Out[6]: '1.12.0.dev0+9506037'
```

Perhaps a bit OT, but I'm curious whether @mrocklin and @shoyer actually consider memmaps to be the best approach for lazily reading binary data with dask / xarray.

See https://github.com/numpy/numpy/pull/7406

The only potential problem I see with just assuming `.filename = None` means "not a real memmap" is that indexed memmaps could potentially be sharing memory with a file, if you used slice operations.

> consider memmaps to be the best approach for lazily reading binary data with dask / xarray.

In my experience, memmaps (e.g., from `scipy.io.netcdf`) have significantly better performance than even non-compressed HDF5 files. So I would say yes, though it could be interesting to compare to zarr.

"Best approach" is hard to define and pretty context specific.  I think
that there are several cases where memmapped files are great.  That being
said, I find myself usually using HDF5.  Echoing @shoyer, I am enthusiastic
about zarr

On Wed, Sep 21, 2016 at 12:19 PM, Ryan Abernathey notifications@github.com
wrote:

> Perhaps a bit OT, but I'm curious whether @mrocklin
> https://github.com/mrocklin and @shoyer https://github.com/shoyer
> actually consider memmaps to be the best approach for lazily reading binary
> data with dask / xarray.
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> https://github.com/dask/dask/issues/1562#issuecomment-248664010, or mute
> the thread
> https://github.com/notifications/unsubscribe-auth/AASszJiKwPzxqLtHXNt8H4EamAO_1aVpks5qsVkEgaJpZM4KC5sI
> .

To clarify, I am talking about a situation where the binary files are already generated and written to disk, in this case by a legacy model with an old-fashioned I/O approach.

If it's already on disk in a memory-mappable format, it's very hard to
imagine beating np.memmap.

On Wed, Sep 21, 2016 at 10:13 AM, Ryan Abernathey notifications@github.com
wrote:

> To clarify, I am talking about a situation where the binary files are
> already generated and written to disk, in this case by a legacy model with
> an old-fashioned I/O approach.
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> https://github.com/dask/dask/issues/1562#issuecomment-248679609, or mute
> the thread
> https://github.com/notifications/unsubscribe-auth/ABKS1h4kti86zJq-Cm9P3FKjZZ6NIuNbks5qsWXPgaJpZM4KC5sI
> .


Resolving patch:
diff --git a/dask/base.py b/dask/base.py
--- a/dask/base.py
+++ b/dask/base.py
@@ -318,7 +318,7 @@ def normalize_dataframe(df):
     def normalize_array(x):
         if not x.shape:
             return (str(x), x.dtype)
-        if hasattr(x, 'mode') and hasattr(x, 'filename'):
+        if hasattr(x, 'mode') and getattr(x, 'filename', None):
             return x.filename, os.path.getmtime(x.filename), x.dtype, x.shape
         if x.dtype.hasobject:
             try:


Test patch:
diff --git a/dask/tests/test_base.py b/dask/tests/test_base.py
--- a/dask/tests/test_base.py
+++ b/dask/tests/test_base.py
@@ -121,6 +121,19 @@ def test_tokenize_numpy_memmap():
     assert y != z
 
 
+@pytest.mark.skipif('not np')
+def test_tokenize_numpy_memmap_no_filename():
+    # GH 1562:
+    with tmpfile('.npy') as fn1, tmpfile('.npy') as fn2:
+        x = np.arange(5)
+        np.save(fn1, x)
+        np.save(fn2, x)
+
+        a = np.load(fn1, mmap_mode='r')
+        b = a + a
+        assert tokenize(b) == tokenize(b)
+
+
 def test_normalize_base():
     for i in [1, 1.1, '1', slice(1, 2, 3)]:
         assert normalize_token(i) is i