Repository: dask/distributed

Issue:
dworker should wait for dcenter to come up
if dworker is running before dcenter process has started, worker never connects to the center. Consider:

```
$dworker 10.52.10.8:8787
distributed.nanny - INFO - Nanny starts worker process 10.52.10.8:8788
distributed.worker - INFO - Start worker at             10.52.10.8:8789
distributed.worker - INFO - Waiting to connect to       10.52.10.8:8787
```

and then:

```
$ dcenter 
distributed.dcenter - INFO - Start center at 10.52.10.8:8787
```

it appears that while dcenter process is running after dworker, dworker does not try to connect with the center again.

This is a slight issue when lunching a mesos task with center and worker process; mesos launches all tasks (center and workers) simultaneously, however the cadence of when center and worker task start running may not be in-sync.



Discussion:
Hrm, this is odd.  I actually get correct behavior here

```
mrocklin@notebook:~$ dworker 127.0.0.1:8787
distributed.nanny - INFO - Start Nanny at:             192.168.1.106:36639
distributed.worker - INFO - Start worker at             192.168.1.106:39815
distributed.worker - INFO - Waiting to connect to       127.0.0.1:8787
...wait...
distributed.worker - INFO - Registered with center at:  127.0.0.1:8787
distributed.nanny - INFO - Nanny 192.168.1.106:36639 starts worker process 192.168.1.106:39815

```

```
mrocklin@notebook:~$ dcenter 
distributed.dcenter - INFO - Start center at 192.168.1.106:8787
distributed.core - INFO - Connection from 127.0.0.1:51274 to Center
distributed.center - INFO - Register ('192.168.1.106', 39815)
```

By default The worker retries every 500 ms until connection with no timeout

https://github.com/blaze/distributed/blob/master/distributed/worker.py#L106-L114

``` python
        while True:
            try:
                resp = yield self.center.register(
                        ncores=self.ncores, address=(self.ip, self.port),
                        nanny_port=self.nanny_port, keys=list(self.data))
                break
            except OSError:
                logger.debug("Unable to register with center.  Waiting")
                yield gen.sleep(0.5)
```

Perhaps the exception type is different for different OS's?  I'm on linux.


Resolving patch:
diff --git a/distributed/worker.py b/distributed/worker.py
--- a/distributed/worker.py
+++ b/distributed/worker.py
@@ -16,12 +16,13 @@
 from tornado.gen import Return
 from tornado import gen
 from tornado.ioloop import IOLoop, PeriodicCallback
+from tornado.iostream import StreamClosedError
 
 from .client import _gather, pack_data, gather_from_workers
 from .compatibility import reload
 from .core import rpc, Server, pingpong
 from .sizeof import sizeof
-from .utils import funcname, get_ip, ignoring
+from .utils import funcname, get_ip
 
 _ncores = ThreadPool()._processes
 
@@ -109,7 +110,7 @@ def _start(self, port=0):
                         ncores=self.ncores, address=(self.ip, self.port),
                         nanny_port=self.nanny_port, keys=list(self.data))
                 break
-            except OSError:
+            except (OSError, StreamClosedError):
                 logger.debug("Unable to register with center.  Waiting")
                 yield gen.sleep(0.5)
         assert resp == b'OK'


Test patch:
diff --git a/distributed/tests/test_worker.py b/distributed/tests/test_worker.py
--- a/distributed/tests/test_worker.py
+++ b/distributed/tests/test_worker.py
@@ -2,17 +2,16 @@
 import os
 import shutil
 import sys
-from time import sleep
 
 from distributed.center import Center
-from distributed.core import read, write, rpc, connect
+from distributed.core import rpc
 from distributed.sizeof import sizeof
-from distributed.utils import ignoring
 from distributed.worker import Worker
 from distributed.utils_test import loop, _test_cluster, inc
+import pytest
 
 from tornado import gen
-from tornado.ioloop import IOLoop
+from tornado.ioloop import TimeoutError
 
 
 def test_worker_ncores():
@@ -237,6 +236,20 @@ def f():
         assert w.port > 1024
 
     loop.run_sync(f)
+
+@pytest.mark.slow
+def test_worker_waits_for_center_to_come_up(loop):
+    @gen.coroutine
+    def f():
+        w = Worker('127.0.0.1', 8007, ip='127.0.0.1')
+        yield w._start()
+
+    try:
+        loop.run_sync(f, timeout=4)
+    except TimeoutError:
+        pass
+
+
 """
 
 def test_close():