Repository: dask/distributed

Issue:
Consider making Scheduler non-reliant on modules used by client/worker
I currently have a process where the client and the workers all have access to some custom Python modules, but the scheduler runs on a clean box with nothing by distributed and its requirements installed.

This works fine, except when a worker raises an exception in a class defined in the custom module. In this case, I get:

```
No module named custom
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/distributed/scheduler.py", line 1432, in handle_worker
    worker, clean_exception(**msg)[1])
  File "/usr/local/lib/python2.7/dist-packages/distributed/core.py", line 711, in clean_exception
    exception = protocol.pickle.loads(exception)
  File "/usr/local/lib/python2.7/dist-packages/distributed/protocol/pickle.py", line 59, in loads
    return pickle.loads(x)
ImportError: No module named custom    
```




Discussion:
This is normally the case.  It looks like we fail to do this when the worker fails unexpectedly, in which case we do deserialize the message for logging.  It seems reasonable to me to either wrap this in a try-except, or better yet send a message from the worker to the scheduler in this case with a pre-formed error message.  

This would probably be best handled by returning a message like 

```python
except Exception as e:
    raise gen.Return({'status': 'error', 'message': str(e)})
```

in the `compute_stream` method.
I've not been through this bit of the code: would your second solution preserve the client receiving the traceback?
No, it would not.  If you wanted to preserve that then you would need to
create it explicitly on the worker and send it as text.

For this I think that you want distributed.utils.get_traceback, and
traceback.format_tb

On Fri, Sep 15, 2017 at 9:05 AM, rbubley <notifications@github.com> wrote:

> I've not been through this bit of the code: would your second solution
> preserve the client receiving the traceback?
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/distributed/issues/1396#issuecomment-329777060>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AASszF-2w1Gc7ZaPNO32y-IdeS5l07vsks5sinYHgaJpZM4PYx5E>
> .
>

Thinking about this, errors in work that the client sends to the worker should be between the worker and the client; the scheduler can reasonably log that there was an error, but shouldn't be expected to know what the error was. 

Thinking this way, your first proposal, of try-except, seems better.

It could just be:
```python
                            try:
                                logger.error("error from worker %s: %s",
                                         worker, clean_exception(**msg)[1])
                            except ImportError:
                                logger.error("error from worker %s",
                                         worker)
```

Normal task exceptions do get sent directly to the user as you suggest.

What you experienced wasn't an error in a client's task, this is an error
in the execution of the worker code itself.  Something serious happened
here that was the fault of the Dask system, not of the user.



On Fri, Sep 15, 2017 at 9:16 AM, rbubley <notifications@github.com> wrote:

> Thinking about this, errors in work that the client sends to the worker
> should be between the worker and the client; the scheduler can reasonably
> log that there was an error, but shouldn't be expected to know what the
> error was.
>
> Thinking this way, your first proposal, of try-except, seems better.
>
> It could just be:
>
>                            try:
>                                 logger.error("error from worker %s: %s",
>                                          worker, clean_exception(**msg)[1])
>                             except ImportError:
>                                 logger.error("error from worker %s",
>                                          worker)
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/distributed/issues/1396#issuecomment-329779649>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AASszBwTCHa5rbSahvV4OezmUa3mcb73ks5sini2gaJpZM4PYx5E>
> .
>

So, some more details...

The client's task raised a ValueError, and then the scheduler hit an error in clean_exception, preventing it from returning proper results to the client.

Client got:
```
>>> resultFuture.status
'error'

>>> resultFuture.exception()
KilledWorker('pricer-a490b119b9d852da1cd913afeeb8826e', 'tls://[REDACTED]')
```
and the client log showed:
```
2017-09-15 10:08:46,157 - ERROR - tornado.application - 59792 - __del__ - Future exception was never retrieved: Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tornado/gen.py", line 1069, in run
    yielded = self.gen.send(value)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/distributed-1.18.3-py2.7.egg/distributed/client.py", line 1218, in wait
    raise AllExit()
AllExit
```

After adding the try-except to the scheduler, all of these problems went away.

Happy to raise a PR for the try-except solution.

Resolving patch:
diff --git a/distributed/scheduler.py b/distributed/scheduler.py
--- a/distributed/scheduler.py
+++ b/distributed/scheduler.py
@@ -1426,11 +1426,12 @@ def handle_worker(self, worker):
                     for msg in msgs:
                         if msg == 'OK':  # from close
                             break
-
                         if 'status' in msg and 'error' in msg['status']:
-                            logger.error("error from worker %s: %s",
+                            try:
+                                logger.error("error from worker %s: %s",
                                          worker, clean_exception(**msg)[1])
-
+                            except Exception:
+                                logger.error("error from worker %s", worker)
                         op = msg.pop('op')
                         if op:
                             self.correct_time_delay(worker, msg)


Test patch: