Repository: dask/dask

Issue:
segmentation fault with dask.dataframe.read_hdf
I continue in my quest to use dask with a large hdf file.

After learning that my original pytables-generated HDF files might not be able to be read by dask.dataframe (because dask uses subselection: pydata/pandas#11188), I tried re-generating the files using the pandas HDFStore interface. (First I had to updated pytables overcome a different bug: pydata/pandas#10672.) (Also, in the meantime, a workaround was found for reading the pytables file into dask: #747, #759.)

Anyway, now I have a pandas-generated HDF file. I can read it with pandas, but with dask I am getting a segfault.

``` python
import pandas as pd
import dask.dataframe as dd
fname = '/data/scratch/rpa/global_lagrangian/run/pandas_floats.h5'
key = '/floats/trajectories'
df = pd.read_hdf(fname, key)
ddf = dd.read_hdf(fname, key)
print(len(df))
print(len(ddf))
```

The first print statement gives `753664023`. The second generates a segfault via the `len` call.

I can still call `ddf.head()` and `ddf.tail()`, but other more complex operations (e.g. `groupby`) fail, presumably due to the same underlying issue.

I am using all the latest masters of dask, pandas, and pytables.



Discussion:
This might be hard to debug without a little more info. Could you try stepping through the computation with pdb to see where the segfault happens?

This would probably require the use of `gdb` rather than `pdb`.

I think that the right thing to do from a dask development perspective is to kick this up to pytables as an issue about thread-safety.  I suspect that if you were to make many calls like `pd.read_hdf(..., start=100000*i, stop=100000*(i+1))` in different threads that this would trigger the issue without dask (this is, in fact, all that dask is doing with `read_hdf`).  I hesitate to dive into the PyTables code personally.

@rabernat are you able to produce a minimum working example of this segfault, preferably with data that we can share?  

If you just want to get things working then I suspect you could turn off threading and things would be fine (if slightly slower)

``` python
import dask
dask.set_options(get=dask.async.get_sync)  # turn off threads globally
```

We could also throw in a `lock=True` keyword argument around the `pd.read_hdf` calls

:+1: for `lock=True`. HDF5 does not support concurrency with threads (this is normally not an issue unless you've enabled compression or are using multiple disks), so we can either add our own lock or push the lock upstream to pytables, but in the end it will all be the same. 

@rabernat can you verify that dd.read_hdf works fine in your example with a single thread?  If so then I'll go ahead and add a lock around the `pd.read_hdf` calls.

``` python
import dask
dask.set_options(get=dask.async.get_sync)  # turn off threads globally
```

Yes, I can verify that dd.read_hdf works with if I first call `dask.set_options(get=dask.async.get_sync)`. (It's is quite slow though--I guess it has to read the whole dataset just figure out how many rows in the table.)

Do you still want me to develop a minimum non-working example? Or is the source of the problem (HDF5 threading issues) sufficiently clear already?


Resolving patch:
diff --git a/dask/dataframe/io.py b/dask/dataframe/io.py
--- a/dask/dataframe/io.py
+++ b/dask/dataframe/io.py
@@ -1,17 +1,19 @@
 from __future__ import division
 
-import pandas as pd
-import numpy as np
+from fnmatch import fnmatch
 from functools import wraps, partial
-import re
-import os
 from glob import glob
 from math import ceil
-from toolz import merge, assoc, dissoc
 from operator import getitem
-from fnmatch import fnmatch
+import os
+import re
+from threading import Lock
 import uuid
 
+import pandas as pd
+import numpy as np
+from toolz import merge, assoc, dissoc
+
 from ..compatibility import BytesIO, unicode, range, apply
 from ..utils import textblock, file_size, get_bom
 from ..base import compute, tokenize
@@ -608,7 +610,7 @@ def to_hdf(df, path_or_buf, key, mode='a', append=False, complevel=0,
 dataset."""
 
 def _read_single_hdf(path, key, start=0, stop=None, columns=None,
-                     chunksize=int(1e6)):
+                     chunksize=int(1e6), lock=None):
     """
     Read a single hdf file into a dask.dataframe. Used for each file in
     read_hdf.
@@ -633,7 +635,8 @@ def get_keys_and_stops(path, key, stop):
                     stops.append(stop)
         return keys, stops
 
-    def one_path_one_key(path, key, start, stop, columns, chunksize):
+
+    def one_path_one_key(path, key, start, stop, columns, chunksize, lock):
         """
         Get the data frame corresponding to one path and one key (which should
         not contain any wildcards).
@@ -645,26 +648,41 @@ def one_path_one_key(path, key, start, stop, columns, chunksize):
                           stop, columns, chunksize))
         name = 'read-hdf-' + token
 
-        dsk = dict(((name, i), (apply, pd.read_hdf,
-                                 (path, key),
-                                 {'start': s, 'stop': s + chunksize,
+        dsk = dict(((name, i), (_pd_read_hdf, path, key, lock,
+                                 {'start': s,
+                                  'stop': s + chunksize,
                                   'columns': columns}))
                     for i, s in enumerate(range(start, stop, chunksize)))
 
         divisions = [None] * (len(dsk) + 1)
         return DataFrame(dsk, name, columns, divisions)
 
+    if lock is True:
+        lock = Lock()
+
     keys, stops = get_keys_and_stops(path, key, stop)
     if (start != 0 or stop is not None) and len(keys) > 1:
         raise NotImplementedError(read_hdf_error_msg)
     from .multi import concat
-    return concat([one_path_one_key(path, k, start, s, columns, chunksize)
+    return concat([one_path_one_key(path, k, start, s, columns, chunksize, lock)
                    for k, s in zip(keys, stops)])
 
 
+def _pd_read_hdf(path, key, lock, kwargs):
+    """ Read from hdf5 file with a lock """
+    if lock:
+        lock.acquire()
+    try:
+        result = pd.read_hdf(path, key, **kwargs)
+    finally:
+        if lock:
+            lock.release()
+    return result
+
+
 @wraps(pd.read_hdf)
 def read_hdf(pattern, key, start=0, stop=None, columns=None,
-             chunksize=1000000):
+             chunksize=1000000, lock=True):
     """
     Read hdf files into a dask dataframe. Like pandas.read_hdf, except it we
     can read multiple files, and read multiple keys from the same file by using
@@ -704,7 +722,8 @@ def read_hdf(pattern, key, start=0, stop=None, columns=None,
         raise NotImplementedError(read_hdf_error_msg)
     from .multi import concat
     return concat([_read_single_hdf(path, key, start=start, stop=stop,
-                                    columns=columns, chunksize=chunksize)
+                                    columns=columns, chunksize=chunksize,
+                                    lock=lock)
                    for path in paths])
 
 


Test patch:
diff --git a/dask/dataframe/tests/test_io.py b/dask/dataframe/tests/test_io.py
--- a/dask/dataframe/tests/test_io.py
+++ b/dask/dataframe/tests/test_io.py
@@ -759,3 +759,14 @@ def test_encoding_gh601():
             d = d.compute()
             d.index = range(len(d.index))
             assert eq(d, a)
+
+
+def test_read_hdf_doesnt_segfault():
+    with tmpfile('h5') as fn:
+        N = 40
+        df = pd.DataFrame(np.random.randn(N, 3))
+        with pd.HDFStore(fn, mode='w') as store:
+            store.append('/x', df)
+
+        ddf = dd.read_hdf(fn, '/x', chunksize=2)
+        assert len(ddf) == N