Repository: dask/distributed

Issue:
Executor.map should yield results not futures
Executor.map returns a list of Futures, but the standard Executor API suggests that it should return an iterable (e.g. generator) of actual results.

I came upon this when trying to demonstrate that Executors allow swapping of IPython/concurrent.futures and distributed Executors, only to discover that distributed's Executor isn't compatible with the rest.



Discussion:
Yes, this is one of our breaks from the concurrent.futures API, see  http://distributed.readthedocs.org/en/latest/related-work.html#id3

I justify this break within the context of distributed computing because moving data is expensive and because distributed supports futures within `submit/map` so `e.map(func, e.map(func, seq))` work the same in both cases.

Thoughts?  Justifiable or no?  Any thoughts on an appropriate compromise? 

I'm honestly not sure. In IPython parallel, I have a similar issue, which I address by `View.map` returning a single AsyncMapResult, which is a _single_ Future, iterating through which yields actual result.

But when I added the Executor layer, I made it a regular iterator yielding results.

I think it's fine to _have_ a map that deviates from concurrent.futures, but to me, part of the point of Executor is the compatible API. In IPython parallel, the result is two `map`s: View.map that behaves how I like, and Executor.map, which is a thin shim around View.map that provides the common Executor API.

Since the difference is tiny:

``` python
class CompatibleExecutor(distributed.Executor):
    """concurrent.futures-compatible Executor"""
    def map(self, *args, **kwargs):
        list_of_futures = super().map(*args, **kwargs)
        for f in list_of_futures:
            yield f.result() # yield results instead of Futures
```

I would either:
1. make this an option on Executor (e.g. `executor.compatibility_mode = True`) (I have no idea what the name should be)
2. include a small compatibility subclass whose goal is to be API-compatible with others, as in the snippet above

The small API-compatibile subclass seems attractive.  It is, as you say, quite easy and doesn't disrupt other work within distributed.


Resolving patch:
diff --git a/distributed/__init__.py b/distributed/__init__.py
--- a/distributed/__init__.py
+++ b/distributed/__init__.py
@@ -5,7 +5,7 @@
 from .diagnostics import progress
 from .utils import sync
 from .nanny import Nanny
-from .executor import Executor, wait, as_completed, default_executor
+from .executor import Executor, CompatibleExecutor, wait, as_completed, default_executor
 from .scheduler import Scheduler
 
 try:
diff --git a/distributed/executor.py b/distributed/executor.py
--- a/distributed/executor.py
+++ b/distributed/executor.py
@@ -805,6 +805,31 @@ def upload_file(self, filename):
             raise result
 
 
+class CompatibleExecutor(Executor):
+    """ A concurrent.futures-compatible Executor
+
+    A subclass of Executor that conforms to concurrent.futures API,
+    allowing swapping in for other Executors.
+    """
+
+    def map(self, func, *iterables, **kwargs):
+        """ Map a function on a sequence of arguments
+
+        Returns
+        -------
+        iter_results: iterable
+            Iterable yielding results of the map.
+
+        See Also
+        --------
+        Executor.map: for more info
+        """
+        list_of_futures = super(CompatibleExecutor, self).map(
+                                func, *iterables, **kwargs)
+        for f in list_of_futures:
+            yield f.result()
+
+
 @gen.coroutine
 def _wait(fs, timeout=None, return_when='ALL_COMPLETED'):
     if timeout is not None:


Test patch:
diff --git a/distributed/tests/test_executor.py b/distributed/tests/test_executor.py
--- a/distributed/tests/test_executor.py
+++ b/distributed/tests/test_executor.py
@@ -7,6 +7,7 @@
 import os
 import shutil
 import sys
+from threading import Thread
 from time import sleep, time
 
 import pytest
@@ -18,8 +19,8 @@
 from distributed import Center, Worker, Nanny
 from distributed.core import rpc
 from distributed.client import WrappedKey
-from distributed.executor import (Executor, Future, _wait, wait, _as_completed,
-        as_completed, tokenize, _global_executor, default_executor)
+from distributed.executor import (Executor, Future, CompatibleExecutor, _wait, wait,
+        _as_completed, as_completed, tokenize, _global_executor, default_executor)
 from distributed.scheduler import Scheduler
 from distributed.sizeof import sizeof
 from distributed.utils import ignoring, sync, tmp_text
@@ -98,6 +99,27 @@ def f(x, y=10):
 
     yield e._shutdown()
 
+@gen_cluster()
+def test_compatible_map(s, a, b):
+    e = CompatibleExecutor((s.ip, s.port), start=False)
+    yield e._start()
+
+    results = e.map(inc, range(5))
+    assert not isinstance(results, list)
+    # Since this map blocks as it waits for results,
+    # waiting here will block the current IOLoop,
+    # which happens to also be running the test Workers.
+    # So wait on the results in a background thread to avoid blocking.
+    f = gen.Future()
+    def wait_on_results():
+        f.set_result(list(results))
+    Thread(target=wait_on_results).start()
+    result_list = yield f
+    # getting map results blocks
+    assert result_list == list(map(inc, range(5)))
+
+    yield e._shutdown()
+
 
 @gen_cluster()
 def test_future(s, a, b):