Repository: dask/distributed

Issue:
Programmatically launched dworkers fail to terminate cleanly
If a dworker is launched with:

``` python
from distributed.cli import dworker
dworker.main.main(args=['127.0.0.1:8786'], prog_name='dworker', standalone_mode=False)
```

Then everything will run fine until an exit is attempted.  If a Ctrl+C is issued, then the following error message is displayed:

``` console
distributed.worker - INFO -       Start worker at:            127.0.0.1:44731
distributed.worker - INFO -              nanny at:            127.0.0.1:43297
distributed.worker - INFO -               http at:            127.0.0.1:33287
distributed.worker - INFO - Waiting to connect to:            127.0.0.1:8786
distributed.worker - INFO -         Registered to:            127.0.0.1:8786
distributed.core - INFO - Connection from 127.0.0.1:45160 to Worker
distributed.core - INFO - Connection from 127.0.0.1:45162 to Worker
distributed.nanny - INFO - Nanny 127.0.0.1:43297 starts worker process 127.0.0.1:44731
^C
distributed.nanny - INFO - Unregister worker 127.0.0.1:44731 from center
distributed.nanny - INFO - Nanny 127.0.0.1:43297 kills worker process 127.0.0.1:44731
distributed.dworker - ERROR - Failed to cleanly kill worker 127.0.0.1:43297
```

Depending on the setup, if another dworker is launched in the same way in the same process, an IOError will be thrown because an I/O loop is already running.  

Setting the kwarg `standalone_mode=True` is not really an option because click will raise a SystemExit (see [the docs](http://click.pocoo.org/5/api/#click.BaseCommand.main)) and take the whole process with it.

Using `make_context()` and `invoke()` or using the callback directly suffer from the the same issues as presented above.

This was discovered while implementing scopatz/xonsh#1125, and so the associated code may be found there.  This is needed because the easiest and most direct way to makes sure that distributed has access to the xonsh builtins is to simply launch dworkers from within a xonsh process.

Thanks in advance!



Discussion:

Resolving patch:
diff --git a/distributed/cli/dworker.py b/distributed/cli/dworker.py
--- a/distributed/cli/dworker.py
+++ b/distributed/cli/dworker.py
@@ -61,7 +61,7 @@ def main(center, host, worker_port, http_port, nanny_port, nthreads, nprocs,
 
     services = {('http', http_port): HTTPWorker}
 
-    loop = IOLoop.current()
+    loop = IOLoop()
     t = Worker if no_nanny else Nanny
     if host is not None:
         ip = socket.gethostbyname(host)
diff --git a/distributed/nanny.py b/distributed/nanny.py
--- a/distributed/nanny.py
+++ b/distributed/nanny.py
@@ -70,7 +70,8 @@ def _kill(self, stream=None, timeout=5):
         if self.process is not None:
             try:
                 result = yield gen.with_timeout(timedelta(seconds=timeout),
-                            self.center.unregister(address=self.worker_address))
+                            self.center.unregister(address=self.worker_address),
+                            io_loop=self.loop)
                 if result not in ('OK', 'already-removed'):
                     logger.critical("Unable to unregister with center %s. "
                             "Nanny: %s, Worker: %s", result, self.address_tuple,


Test patch: