Repository: dask/dask

Issue:
Lock on bcolz
Hello,

Still playing with this package.

I have a question about the use of threads/locks on dataframe imports and bcolz. 

Take this section of code:

```
def locked_df_from_ctable(*args, **kwargs):
    with lock:
        result = dataframe_from_ctable(*args, **kwargs)
    return result
```

I _think_ what's happening here is that the lock prevents the dask threads from reading more than one bcolz file at a time. In my particular case, I have a file of about 7 million records in a bcolz dataset. When I just got rid of the lock... (i.e., got rid of the `with lock` statement and un-indented `result = ...`, the processing time was cut from around 90 seconds to 50 seconds. CPU usage never exceed 50%. (I presume the bottleneck was hard drive read speeds). 

So what's the purpose of this lock? Is it necessary in some legacy packages? I ask because I'd like to make a PR to get rid of it. In my test, I was able to just drop it and I got the same result.

Thanks. 



Discussion:
Historically bcolz didn't support safe concurrent reads, notably because of Blosc.  However, blosc has been seeing a fair amount of activity so we should check with the maintainers.  

@FrancescAlted, does BColz safely support multiple concurrent reads?

Thanks for the super fast response. 

Another question: Essentially, I'm building a tool that needs to read a large-ish file. I experimented with both bcolz and castra and found bcolz had better performance than castra with my data--even though bcolz had all the locking going on and castra doesn't. 

What I'm writing is pretty agnostic between the different dataframe input formats, but is there a preference as for what is the best tool going forward? I assumed bcolz just because castra describes itself as "experimental". 

Thanks.

BColz is definitely more mature.  Castra would handle categoricals and valued-indexes better.  If BColz fits your use case I would definitely use it.

Also, per this, looks like blosc is now thread-safe since version 1.1. 

https://github.com/Blosc/python-blosc/wiki/Release-notes

If you are open to a PR, let me know. Happy to build it with your blessing and guidance. 

Looks like you don't need my guidance :)

I still recommend that we wait on a response from @FrancescAlted .  It may be that blosc lost thread-safe reading after 1.1.  It may also be that the version that ships with bcolz is different than what's generally released.  The history is a bit complex here.

Ok. 

What I mean is that I don't want to do something that will break backwards compatibility, so we'd have to be cautious about how its written. 

Hi there,

I spent some time last year looking into bcolz, dask and thread safety.
It's a bit complicated, but I'll try to distil the main points.

I believe that in general bcolz is currently not thread safe. More
discussion and reasoning is on this PR:
https://github.com/Blosc/bcolz/pull/286.

If you are only ever reading bcolz ctables from dask then it _might_ be ok
not to lock a ctable. However, the way bcolz currently uses blosc, blosc
maintains a global lock, and so you can only ever read or write to a single
carray anyway. I.e., if dask did not use a read lock, you still would not
see any performance improvements.

Later versions of blosc (>1.7.0) have alternate "contextual" functions that
avoid the global lock. These in principle would allow a multithreaded
version of bcolz to be developed. I did some work towards this on the above
PR.

However, in the end I decided to implement something from scratch, because
my use cases involve multidimensional arrays and were pulling in a
different direction, see https://github.com/alimanfoo/zarr. In zarr I've
experimented with chunk-level locking, which should in principle allow
concurrent reads or concurrent writes to the same chunked array. However,
zarr only provides an array class and does not provide a table (dataframe)
class, so for your use case it may not be suitable.

Hth.

cc @FrancescAlted.

On Tuesday, March 8, 2016, thequackdaddy notifications@github.com wrote:

> Ok.
> 
> What I mean is that I don't want to do something that will break backwards
> compatibility, so we'd have to be cautious about how its written.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/dask/dask/issues/1033#issuecomment-193530578.

## 

Alistair Miles
Head of Epidemiological Informatics
Centre for Genomics and Global Health http://cggh.org
The Wellcome Trust Centre for Human Genetics
Roosevelt Drive
Oxford
OX3 7BN
United Kingdom
Email: alimanfoo@googlemail.com alimanfoo@gmail.com
Web: http://purl.org/net/aliman
Twitter: https://twitter.com/alimanfoo
Tel: +44 (0)1865 287721

Hi guys, thanks for bringing this to my attention.  Yeah, things are a bit complex here (I have to admit that I sometimes get confused too), so for N-time, I will try to clarify some points here.

C-Blosc 1.x (the library that is used internally by bcolz, and not python-blosc as some seem to suggest) series underwent a series of changes regarding thread safety.  The timeline is something like:

1) C-Blosc uses internal threads since 1.0, but that created havoc with threaded callers.

2) Starting with C-Blosc 1.2, a global internal lock was added.  From this version on, only one thread of the caller application was able to to run C-Blosc simultaneously (although C-Blosc still was able to run different threads internally).

3) In C-Blosc 1.5, the global lock was removed again and two new `blosc_compress_ctx()` and `blosc_decompress_ctx()` were added.  So, from 1.5 on, using these calls is the correct way to call C-Blosc from multi-threaded applications.

That's basically it regarding the C-Blosc multithreading evolution.  Now, it turns out that, sometime shortly after 1.5 (or even in 1.5), we introduced code that was not compatible with MS VS2008 (needed for compatibility with Python 2.6 and 2.7 on Windows).  For more info about why VS2008 is creating me trouble, see https://groups.google.com/forum/#!topic/bcolz/58qHe6gv0iU.  Frankly, I was sending this message in the hope that someone would provide a backport of recent C-Blosc to VS2008, but unfortunately that did not happen.

So we are in a position now that C-Blosc 1.5 or higher cannot be included in bcolz.  This is why I'm maintaining a [C-Blosc 1.4.x branch](https://github.com/Blosc/c-blosc/tree/blosc-1.4.x), which is the one that I use for applications that require support for Python 2 on Windows.  In fact, I recently downgraded C-Blosc inside bcolz master to 1.4.4 precisely for this reason.

Incidentally, this discussion reminded me that it would be useful to have a bcolz 1.0.0 so that people can be sure that the format is not going to change for a long while, and this is why I recently announced bcolz 1.0.0 RC1 with C-Blosc 1.4.4 on it.  Due to the internal global lock in 1.4.4, I think this is safe to use on multi-threaded environments (so no additional lock would be needed), and the only disadvantage is that multi-threaded apps will inherit this lock and that will create contention (but again, bcolz can still use internal C-Blosc threads for accelerating compression/decompression).

Anyways, hope this helps.  And as always, if anybody is unhappy with this situation I am open to discuss on pull requests.  Thanks.

Thanks Francesc. I think it's worth adding that just because c-blosc
maintains a global lock, this does _not_ ensure that bcolz is safe to use
in a multi-threaded application. Consider the worst case where one Python
thread is attempting to read and another write the same bcolz carray. While
each call to a c-blosc function will be mutually exclusive, at a higher
level crazy things could happen, e.g., one thread could completely erase a
carray's data while another thread was half-way through reading it. The GIL
does _not_ prevent this sort of thing happening.

It _may_ be safe for multiple Python threads to read from the same bcolz
carray, although again there may be unexpected things happening at the
level above c-blosc, so I would not want to put any money on it. But in any
case there would be no point, because there would be no performance gain,
because c-blosc maintains a global lock.

In general I think the safest thing is for multi-threaded applications to
maintain a lock when accessing any bcolz carray or ctable, because there is
nothing to lose and everything to gain.

Hth.

On Tue, Mar 8, 2016 at 2:20 PM, FrancescAlted notifications@github.com
wrote:

> Hi guys, thanks for bringing this to my attention. Yeah, things are a bit
> complex here (I have to admit that I sometimes get confused too), so for
> N-time, I will try to clarify some points here.
> 
> C-Blosc 1.x (the library that is used internally by bcolz, and not
> python-blosc as some seem to suggest) series underwent a series of changes
> regarding thread safety. The timeline is something like:
> 
> 1) C-Blosc uses internal threads since 1.0, but that created havoc with
> threaded callers.
> 
> 2) Starting with C-Blosc 1.2, a global internal lock was added. From this
> version on, only one thread of the caller application was able to to run
> C-Blosc simultaneously (although C-Blosc still was able to run different
> threads internally).
> 
> 3) In C-Blosc 1.5, the global lock was removed again and two new
> blosc_compress_ctx() and blosc_decompress_ctx() were added. So, from 1.5
> on, using these calls is the correct way to call C-Blosc from
> multi-threaded applications.
> 
> That's basically it regarding the C-Blosc multithreading evolution. Now,
> it turns out that, sometime shortly after 1.5 (or even in 1.5), we
> introduced code that was not compatible with MS VS2008 (needed for
> compatibility with Python 2.6 and 2.7 on Windows). For more info about why
> VS2008 is creating me trouble, see
> https://groups.google.com/forum/#!topic/bcolz/58qHe6gv0iU. Frankly, I was
> sending this message in the hope that someone would provide a backport of
> recent C-Blosc to VS2008, but unfortunately that did not happen.
> 
> So we are in a position now that C-Blosc 1.5 or higher cannot be included
> in bcolz. This is why I'm maintaining a C-Blosc 1.4.x branch
> https://github.com/Blosc/c-blosc/tree/blosc-1.4.x, which is the one
> that I use for applications that require support for Python 2 on Windows.
> In fact, I recently downgraded C-Blosc inside bcolz master to 1.4.4
> precisely for this reason.
> 
> Incidentally, this discussion reminded me that it would be useful to have
> a bcolz 1.0.0 so that people can be sure that the format is not going to
> change for a long while, and this is why I recently announced bcolz 1.0.0
> RC1 with C-Blosc 1.4.4 on it. Due to the internal global lock in 1.4.4, I
> think this is safe to use on multi-threaded environments (so no additional
> lock would be needed), and the only disadvantage is that multi-threaded
> apps will inherit this lock and that will create contention (but again,
> bcolz can still use internal C-Blosc threads for accelerating
> compression/decompression).
> 
> Anyways, hope this helps. And as always, if anybody is unhappy with this
> situation I am open to discuss on pull requests. Thanks.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/dask/dask/issues/1033#issuecomment-193801398.

## 

Alistair Miles
Head of Epidemiological Informatics
Centre for Genomics and Global Health http://cggh.org
The Wellcome Trust Centre for Human Genetics
Roosevelt Drive
Oxford
OX3 7BN
United Kingdom
Email: alimanfoo@googlemail.com alimanfoo@gmail.com
Web: http://purl.org/net/aliman
Twitter: https://twitter.com/alimanfoo
Tel: +44 (0)1865 287721

Alistair, I think what you are describing is a general problem with writing and reading into the same area and it will happen either with threads or with processes.  But at any rate, I am fine with locks in software on top of bcolz; it is up to the developer to decide this.

Yikes. Feels like I opened a big can-o-worms. 

I can just say this... when I removed the lock statement from the, I got what felt like the same matrix. I didn't check element by element, but the aggregate added up to the same number. Additionally, the wall time was much lower and the dask bokeh profiler do-hickey showed much less time was being used by the `locked_df_from_ctable` function. I'll try to reproduce and send it here. I'll also do some more robust checking of what I did. 

Its also possible that I don't know how to use dask effectively and something I did slowed down the process. 

Thanks. 


Resolving patch:
diff --git a/dask/dataframe/io.py b/dask/dataframe/io.py
--- a/dask/dataframe/io.py
+++ b/dask/dataframe/io.py
@@ -408,7 +408,8 @@ def from_pandas(data, npartitions=None, chunksize=None, sort=True):
     return _Frame(dsk, name, data, divisions)
 
 
-def from_bcolz(x, chunksize=None, categorize=True, index=None, **kwargs):
+def from_bcolz(x, chunksize=None, categorize=True, index=None, lock=lock,
+               **kwargs):
     """ Read dask Dataframe from bcolz.ctable
 
     Parameters
@@ -423,12 +424,17 @@ def from_bcolz(x, chunksize=None, categorize=True, index=None, **kwargs):
         Automatically categorize all string dtypes
     index : string, optional
         Column to make the index
+    lock: bool or Lock
+        Lock to use when reading or False for no lock (not-thread-safe)
 
     See Also
     --------
 
     from_array: more generic function not optimized for bcolz
     """
+    if lock is True:
+        lock = Lock()
+
     import dask.array as da
     import bcolz
 
@@ -459,10 +465,10 @@ def from_bcolz(x, chunksize=None, categorize=True, index=None, **kwargs):
     new_name = 'from_bcolz-' + token
 
     dsk = dict(((new_name, i),
-                (locked_df_from_ctable,
+                (dataframe_from_ctable,
                  x,
                  (slice(i * chunksize, (i + 1) * chunksize),),
-                 columns, categories))
+                 columns, categories, lock))
                for i in range(0, int(ceil(len(x) / chunksize))))
 
     result = DataFrame(dsk, new_name, columns, divisions)
@@ -477,7 +483,7 @@ def from_bcolz(x, chunksize=None, categorize=True, index=None, **kwargs):
         return result
 
 
-def dataframe_from_ctable(x, slc, columns=None, categories=None):
+def dataframe_from_ctable(x, slc, columns=None, categories=None, lock=lock):
     """ Get DataFrame from bcolz.ctable
 
     Parameters
@@ -520,29 +526,30 @@ def dataframe_from_ctable(x, slc, columns=None, categories=None):
         stop = slc[0].stop if slc[0].stop < len(x) else len(x)
     idx = pd.Index(range(start, stop))
 
-    if isinstance(x, bcolz.ctable):
-        chunks = [x[name][slc] for name in columns]
-        if categories is not None:
-            chunks = [pd.Categorical.from_codes(np.searchsorted(categories[name],
-                                                                chunk),
-                                                categories[name], True)
-                       if name in categories else chunk
-                       for name, chunk in zip(columns, chunks)]
-        return pd.DataFrame(dict(zip(columns, chunks)), columns=columns,
-                            index=idx)
-
-    elif isinstance(x, bcolz.carray):
-        chunk = x[slc]
-        if categories is not None and columns and columns in categories:
-            chunk = pd.Categorical.from_codes(
-                        np.searchsorted(categories[columns], chunk),
-                        categories[columns], True)
-        return pd.Series(chunk, name=columns, index=idx)
-
-
-def locked_df_from_ctable(*args, **kwargs):
-    with lock:
-        result = dataframe_from_ctable(*args, **kwargs)
+    if lock:
+        lock.acquire()
+    try:
+        if isinstance(x, bcolz.ctable):
+            chunks = [x[name][slc] for name in columns]
+            if categories is not None:
+                chunks = [pd.Categorical.from_codes(
+                                    np.searchsorted(categories[name], chunk),
+                                    categories[name], True)
+                           if name in categories else chunk
+                           for name, chunk in zip(columns, chunks)]
+            result = pd.DataFrame(dict(zip(columns, chunks)), columns=columns,
+                                index=idx)
+
+        elif isinstance(x, bcolz.carray):
+            chunk = x[slc]
+            if categories is not None and columns and columns in categories:
+                chunk = pd.Categorical.from_codes(
+                            np.searchsorted(categories[columns], chunk),
+                            categories[columns], True)
+            result = pd.Series(chunk, name=columns, index=idx)
+    finally:
+        if lock:
+            lock.release()
     return result
 
 


Test patch:
diff --git a/dask/dataframe/tests/test_io.py b/dask/dataframe/tests/test_io.py
--- a/dask/dataframe/tests/test_io.py
+++ b/dask/dataframe/tests/test_io.py
@@ -7,6 +7,7 @@
 import dask
 from operator import getitem
 import pytest
+from threading import Lock
 from toolz import valmap
 import tempfile
 import shutil
@@ -323,6 +324,23 @@ def test_from_bcolz():
            sorted(dsk)
 
 
+def test_from_bcolz_no_lock():
+    bcolz = pytest.importorskip('bcolz')
+    locktype = type(Lock())
+
+    t = bcolz.ctable([[1, 2, 3], [1., 2., 3.], ['a', 'b', 'a']],
+                     names=['x', 'y', 'a'], chunklen=2)
+    a = dd.from_bcolz(t, chunksize=2)
+    b = dd.from_bcolz(t, chunksize=2, lock=True)
+    c = dd.from_bcolz(t, chunksize=2, lock=False)
+    eq(a, b)
+    eq(a, c)
+
+    assert not any(isinstance(item, locktype)
+                   for v in c.dask.values()
+                   for item in v)
+
+
 def test_from_bcolz_filename():
     bcolz = pytest.importorskip('bcolz')