Repository: dask/distributed

Issue:
Better handling of tasks that cause the worker to exit
The following seems to hang forever:

``` python
>>> from distributed import Executor
>>> import sys
>>> e = Executor('localhost:8786')
>>> e.submit(sys.exit, 1).result()
```

in the background this causes the dscheduler logs to output many errors such as:

```
distributed.scheduler - ERROR - Stream is closed
Traceback (most recent call last):
  File "/home/ogrisel/code/distributed/distributed/scheduler.py", line 1545, in heartbeat
    io_loop=self.loop)
  File "/home/ogrisel/.virtualenvs/py35/lib/python3.5/site-packages/tornado/gen.py", line 1008, in run
    value = future.result()
  File "/home/ogrisel/.virtualenvs/py35/lib/python3.5/site-packages/tornado/concurrent.py", line 232, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/home/ogrisel/.virtualenvs/py35/lib/python3.5/site-packages/tornado/gen.py", line 1014, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/ogrisel/code/distributed/distributed/core.py", line 439, in send_recv_from_rpc
    result = yield send_recv(stream=stream, op=key, **kwargs)
  File "/home/ogrisel/.virtualenvs/py35/lib/python3.5/site-packages/tornado/gen.py", line 1008, in run
    value = future.result()
  File "/home/ogrisel/.virtualenvs/py35/lib/python3.5/site-packages/tornado/concurrent.py", line 232, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/home/ogrisel/.virtualenvs/py35/lib/python3.5/site-packages/tornado/gen.py", line 1014, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/ogrisel/code/distributed/distributed/core.py", line 333, in send_recv
    response = yield read(stream)
  File "/home/ogrisel/.virtualenvs/py35/lib/python3.5/site-packages/tornado/gen.py", line 1008, in run
    value = future.result()
  File "/home/ogrisel/.virtualenvs/py35/lib/python3.5/site-packages/tornado/concurrent.py", line 232, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/home/ogrisel/.virtualenvs/py35/lib/python3.5/site-packages/tornado/gen.py", line 1014, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/ogrisel/code/distributed/distributed/core.py", line 241, in read
    n_frames = yield stream.read_bytes(8)
  File "/home/ogrisel/.virtualenvs/py35/lib/python3.5/site-packages/tornado/gen.py", line 1008, in run
    value = future.result()
  File "/home/ogrisel/.virtualenvs/py35/lib/python3.5/site-packages/tornado/concurrent.py", line 232, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
tornado.iostream.StreamClosedError: Stream is closed
```

I think each worker process should have a unique `uuid4` that is passed as a metadata to any Future object returned by those workers and the nannies should report to the scheduler the `uuid4` of any process that crashed when executing a task so that the scheduler can ban those tasks and update the Future objects to mark them as failed with an informative error message instead of hanging for ever.



Discussion:
A similar issue happens if the submitted function causes a segfault:

``` python
>>> from distributed import Executor
>>> from faulthandler import _sigsegv
>>> e.submit(_sigsegv).result()
```

I assume the same would also happen if the submitted task causes the worker process to get killed by the OOM killer of the operating system.

I agree that we should definitely improve cases like these.

Workers _do_ have a unique ID (either uuid or user supplied) and the scheduler does know what workers have what data.  

Banning tasks on worker failure is a bit ambiguous, we would probably only want to ban after multiple workers have died while trying to compute the task.  Otherwise we destroy resilience to worker failure.

I missed the ID when glancing at the code. Indeed trying a limited number of times before reporting the failure to the caller is a good idea.

This would probably be an extra dictionary on the scheduler, something like the following in `__init__`:

```
self.failures = defaultdict(lambda: 0)
```

We would increment this for all tasks in `self.processing[address]` when we call `remove_worker` _if_ we call it a bad situation (we should probably add a kwarg to `remove_worker` if it is called unexpectedly.)  If a task's failure count grows to some large number (perhaps 3) then we `report` a message down to the client that it failed, perhaps with some custom exception.

I can do this late next week.  I also think it'd be a decent task for you if you're interested in getting into how the scheduler works and if you have spare time (which is, of course, unlikely.)  This touches many pieces of the scheduler, but in a light way.  It would be good to have more developers exposed to how the scheduler works internally.

IPython's scheduler has this notion:
- A task has a `retries` property (default: 0)
- If a task fails and has retries left, it is assigned to a new worker (if it failed for other reasons and the worker didn't die, it will never be retried on that worker)
- If an engine dies while running a task, the scheduler injects an artificial "EngineDied" exception and treats that as a regular failure.
- When retries are exhausted, propagate the exception to the user

So the default behavior is that if a worker dies, the user sees that bubble up as an EngineDied exception. If retries is given, then a truly bad task that always segfaults will take down `retries` workers before showing the exception to the user. Under no circumstances should engine death result in hangs for the client.

Unfortunately I don't think I will have the time to dive into the code in the coming weeks.

@minrk if retries is given and the exception is more mundane, like `ZeroDivisionError`, will IPyParallel still retry?

IPython will, yes, but I'm not sure it should. Engine death is the only case I have encountered in the real world where I have seen a task that should be retried, so handling only that makes the most sense to me, at least as a starting point. I thought some things like temporary resource availability issues might come up, but they don't seem to, so I would probably hold off on supporting that until someone shows up with a real need for it (and a PR!).


Resolving patch:
diff --git a/distributed/scheduler.py b/distributed/scheduler.py
--- a/distributed/scheduler.py
+++ b/distributed/scheduler.py
@@ -4,6 +4,7 @@
 from datetime import datetime, timedelta
 import logging
 import math
+import pickle
 import random
 import socket
 from time import time
@@ -35,6 +36,7 @@
 logger = logging.getLogger(__name__)
 
 BANDWIDTH = 100e6
+ALLOWED_FAILURES = 3
 
 
 class Scheduler(Server):
@@ -127,6 +129,8 @@ class Scheduler(Server):
     * **loose_retrictions:** ``{key}``:
         Set of keys for which we are allow to violate restrictions (see above)
         if not valid workers are present.
+    * **suspicious_tasks:** ``{key: int}``
+        Number of times a task has been involved in a worker failure
     * **keyorder:** ``{key: tuple}``:
         A score per key that determines its priority
     * **scheduler_queues:** ``[Queues]``:
@@ -184,6 +188,7 @@ def __init__(self, center=None, loop=None,
         self.task_duration = {prefix: 0.00001 for prefix in fast_task_prefixes}
         self.restrictions = dict()
         self.loose_restrictions = set()
+        self.suspicious_tasks = defaultdict(lambda: 0)
         self.stacks = dict()
         self.waiting = dict()
         self.waiting_data = dict()
@@ -750,6 +755,17 @@ def update_data(self, who_has=None, nbytes=None, client=None):
             if key not in self.dependencies:
                 self.dependencies[key] = set()
 
+    def mark_task_killed_worker(self, key=None, worker=None):
+        """ Mark that is likely killing workers """
+        self.exceptions[key] = pickle.dumps(KilledWorker(key, worker))
+        self.tracebacks[key] = None
+        self.mark_failed(key, key)
+        for plugin in self.plugins[:]:
+            try:
+                plugin.task_erred(self, key=key, worker=worker,
+                        exception=exception, traceback=traceback, **kwargs)
+            except Exception as e:
+                logger.exception(e)
 
     def mark_task_erred(self, key=None, worker=None,
                         exception=None, traceback=None, **kwargs):
@@ -765,7 +781,6 @@ def mark_task_erred(self, key=None, worker=None,
             self.tracebacks[key] = traceback
             self.mark_failed(key, key)
             self.maybe_idle.add(worker)
-            # self.ensure_occupied(worker)
             for plugin in self.plugins[:]:
                 try:
                     plugin.task_erred(self, key=key, worker=worker,
@@ -953,11 +968,15 @@ def remove_worker(self, stream=None, address=None):
                 self.saturated.remove(address)
 
             in_flight = set(self.processing.pop(address))
-            for k in in_flight:
+            for k in list(in_flight):
+                self.suspicious_tasks[k] += 1
                 s = self.rprocessing[k]
                 s.remove(address)
                 if not s:
                     del self.rprocessing[k]
+                if self.suspicious_tasks[k] > ALLOWED_FAILURES:
+                    self.mark_task_killed_worker(key=k, worker=address)
+                    in_flight.remove(k)
             in_flight |= set(self.stacks.pop(address))
             del self.occupancy[address]
             in_flight = {k for k in in_flight if k in self.tasks}
@@ -1252,6 +1271,8 @@ def forget(self, key):
                     del self.waiting[key]
                 if key in self.waiting_data:
                     del self.waiting_data[key]
+                if key in self.suspicious_tasks:
+                    del self.suspicious_tasks[key]
                 for w in self.rprocessing.pop(key, ()):
                     self.occupancy[w] -= self.processing[w].pop(key)
 
@@ -2133,3 +2154,7 @@ def check_key(key):
 
 
 fast_task_prefixes = {'rechunk-split'}
+
+
+class KilledWorker(Exception):
+    pass


Test patch:
diff --git a/distributed/tests/test_executor.py b/distributed/tests/test_executor.py
--- a/distributed/tests/test_executor.py
+++ b/distributed/tests/test_executor.py
@@ -23,7 +23,7 @@
 from distributed.executor import (Executor, Future, CompatibleExecutor, _wait,
         wait, _as_completed, as_completed, tokenize, _global_executor,
         default_executor, _first_completed, ensure_default_get, futures_of)
-from distributed.scheduler import Scheduler
+from distributed.scheduler import Scheduler, KilledWorker
 from distributed.sizeof import sizeof
 from distributed.utils import sync, tmp_text
 from distributed.utils_test import (cluster, slow, slowinc, slowadd, randominc,
@@ -3003,6 +3003,18 @@ def test_get_stacks_processing(e, s, a, b):
     assert processing == valmap(list, s.processing)
 
 
+@gen_cluster(executor=True, Worker=Nanny)
+def test_bad_tasks_fail(e, s, a, b):
+    f = e.submit(sys.exit, 1)
+    with pytest.raises(KilledWorker):
+        yield f._result()
+
+    start = time()
+    while len(s.ncores) < 2:
+        yield gen.sleep(0.01)
+        assert time() < start + 5
+
+
 def test_get_stacks_processing_sync(loop):
     with cluster() as (s, [a, b]):
         with Executor(('127.0.0.1', s['port']), loop=loop) as e: