Repository: dask/distributed

Issue:
Joblib interface
Should we support a `joblib interface` alongside the `concurrent.futures` interface?  This would be for simple embarrassingly parallel computation.  It would require us to think about auto-batching long sequences of small inputs to ensure we tailored batch size to hit a nice frequency of output.  This would add novel capability to existing joblib users in two ways:
1.  They could nest several functions that call out to joblib smoothly (this is a current pain point)
2.  They would get distributed computation

This would be a nice way to support existing codebases within libraries like scikit-learn.



Discussion:
Now that the backend refactoring of joblib has been merged in master I would be in favor of shipping a distributed backend implementation instead.

Here is an example of scikit-learn using distributed via this backend:

https://github.com/ogrisel/docker-distributed/blob/master/examples/sklearn_parameter_search.ipynb

the code of the backend is there:

https://github.com/ogrisel/docker-distributed/blob/master/examples/distributed_joblib_backend.py

Was the joblib-distributed backend used in that notebook?  It looks like you were submitting tasks manually.

Oops, sorry, that's was the wrong example. Here is the example with the joblib backend:

https://github.com/ogrisel/docker-distributed/blob/master/examples/sklearn_parameter_search_joblib.ipynb

That's really cool.  Did you notice a speedup?  Its hard to compare the numbers in the notebook directly.

I'll go over the joblib `DistributedBackend` sometime today, test it a bit more extensively and submit a PR, probably sometime tomorrow (traveling a bit today.)


Resolving patch:
diff --git a/distributed/joblib.py b/distributed/joblib.py
new file mode 100644
--- /dev/null
+++ b/distributed/joblib.py
@@ -0,0 +1,39 @@
+from __future__ import print_function, division, absolute_import
+
+from joblib._parallel_backends import ParallelBackendBase, AutoBatchingMixin
+from joblib.parallel import register_parallel_backend
+from tornado import gen
+
+from .executor import Executor, _wait
+
+
+class DistributedBackend(ParallelBackendBase, AutoBatchingMixin):
+    MIN_IDEAL_BATCH_DURATION = 0.2
+    MAX_IDEAL_BATCH_DURATION = 1.0
+
+    def __init__(self, scheduler_host='127.0.0.1:8786', loop=None):
+        self.executor = Executor(scheduler_host, loop=loop)
+
+    def configure(self, n_jobs=1, parallel=None, **backend_args):
+        return self.effective_n_jobs(n_jobs)
+
+    def effective_n_jobs(self, n_jobs=1):
+        return sum(self.executor.ncores().values())
+
+    def apply_async(self, func, *args, **kwargs):
+        callback = kwargs.pop('callback', None)
+        kwargs['pure'] = False
+        future = self.executor.submit(func, *args, **kwargs)
+
+        @gen.coroutine
+        def callback_wrapper():
+            result = yield _wait([future])
+            callback(result)  # gets called in separate thread
+
+        self.executor.loop.add_callback(callback_wrapper)
+
+        future.get = future.result  # monkey patch to achieve AsyncResult API
+        return future
+
+
+register_parallel_backend('distributed', DistributedBackend)
diff --git a/distributed/utils.py b/distributed/utils.py
--- a/distributed/utils.py
+++ b/distributed/utils.py
@@ -200,6 +200,8 @@ def key_split(s):
     'hello-world'
     >>> key_split('ae05086432ca935f6eba409a8ecd4896')
     'data'
+    >>> key_split('<module.submodule.myclass object at 0xdaf372')
+    'myclass'
     >>> key_split(None)
     'Other'
     """
@@ -218,6 +220,8 @@ def key_split(s):
         if len(result) == 32 and re.match(r'[a-f0-9]{32}', result):
             return 'data'
         else:
+            if result[0] == '<':
+                result = result.strip('<>').split()[0].split('.')[-1]
             return result
     except:
         return 'Other'


Test patch:
diff --git a/distributed/tests/test_joblib.py b/distributed/tests/test_joblib.py
new file mode 100644
--- /dev/null
+++ b/distributed/tests/test_joblib.py
@@ -0,0 +1,33 @@
+from __future__ import print_function, division, absolute_import
+
+import pytest
+pytest.importorskip('joblib')
+from random import random
+
+from joblib import parallel_backend, Parallel, delayed
+
+import distributed.joblib
+from distributed.utils_test import inc, cluster, loop
+
+def test_simple(loop):
+    with cluster() as (s, [a, b]):
+        with parallel_backend('distributed', loop=loop,
+                scheduler_host=('127.0.0.1', s['port'])):
+
+            seq = Parallel()(delayed(inc)(i) for i in range(10))
+            assert seq == [inc(i) for i in range(10)]
+
+            seq = Parallel()(delayed(inc)(i) for i in range(10))
+            assert seq == [inc(i) for i in range(10)]
+
+
+def random2():
+    return random()
+
+def test_dont_assume_function_purity(loop):
+    with cluster() as (s, [a, b]):
+        with parallel_backend('distributed', loop=loop,
+                scheduler_host=('127.0.0.1', s['port'])):
+
+            x, y = Parallel()(delayed(random2)() for i in range(2))
+            assert x != y