Repository: dask/distributed

Issue:
dask-worker timeout should exit with non-zero status code?
Working on dask-kubernetes, I noticed that my pods were succeeding, when they were timing out trying to connect to a scheduler.

In this example, I don't have a scheduler running at this address

```
bash-5.0$ dask-worker 192.168.7.26:8786 --death-timeout=1
2019-06-18 14:24:19,779 distributed.nanny[41007] INFO         Start Nanny at: 'tcp://192.168.7.20:64424'
2019-06-18 14:24:20,790 distributed.nanny[41007] INFO Closing Nanny at 'tcp://192.168.7.20:64424'
2019-06-18 14:24:21,330 distributed.worker[41030] INFO       Start worker at:   tcp://192.168.7.20:64425
2019-06-18 14:24:21,331 distributed.worker[41030] INFO          Listening to:   tcp://192.168.7.20:64425
2019-06-18 14:24:21,331 distributed.worker[41030] INFO          dashboard at:         192.168.7.20:64426
2019-06-18 14:24:21,332 distributed.worker[41030] INFO Waiting to connect to:    tcp://192.168.7.26:8786
2019-06-18 14:24:21,332 distributed.worker[41030] INFO -------------------------------------------------
2019-06-18 14:24:21,333 distributed.worker[41030] INFO               Threads:                          8
2019-06-18 14:24:21,333 distributed.worker[41030] INFO                Memory:                   17.18 GB
2019-06-18 14:24:21,334 distributed.worker[41030] INFO       Local Directory: /Users/taugspurger/worker-a2hgbeb6
2019-06-18 14:24:21,334 distributed.worker[41030] INFO -------------------------------------------------
2019-06-18 14:24:21,335 distributed.worker[41030] INFO Stopping worker at tcp://192.168.7.20:64425
2019-06-18 14:24:21,379 distributed.dask_worker[41007] INFO End worker
bash-5.0$ echo $?
0
```

Would changing that to be non-zero break anything else?


Discussion:
As far as I know that would be totally fine

On Tue, Jun 18, 2019 at 9:25 PM Tom Augspurger <notifications@github.com>
wrote:

> Working on dask-kubernetes, I noticed that my pods were succeeding, when
> they were timing out trying to connect to a scheduler.
>
> bash-5.0$ dask-worker 192.168.7.26:8786 --death-timeout=1
> 2019-06-18 14:24:19,779 distributed.nanny[41007] INFO         Start Nanny at: 'tcp://192.168.7.20:64424'
> 2019-06-18 14:24:20,790 distributed.nanny[41007] INFO Closing Nanny at 'tcp://192.168.7.20:64424'
> 2019-06-18 14:24:21,330 distributed.worker[41030] INFO       Start worker at:   tcp://192.168.7.20:64425
> 2019-06-18 14:24:21,331 distributed.worker[41030] INFO          Listening to:   tcp://192.168.7.20:64425
> 2019-06-18 14:24:21,331 distributed.worker[41030] INFO          dashboard at:         192.168.7.20:64426
> 2019-06-18 14:24:21,332 distributed.worker[41030] INFO Waiting to connect to:    tcp://192.168.7.26:8786
> 2019-06-18 14:24:21,332 distributed.worker[41030] INFO -------------------------------------------------
> 2019-06-18 14:24:21,333 distributed.worker[41030] INFO               Threads:                          8
> 2019-06-18 14:24:21,333 distributed.worker[41030] INFO                Memory:                   17.18 GB
> 2019-06-18 14:24:21,334 distributed.worker[41030] INFO       Local Directory: /Users/taugspurger/worker-a2hgbeb6
> 2019-06-18 14:24:21,334 distributed.worker[41030] INFO -------------------------------------------------
> 2019-06-18 14:24:21,335 distributed.worker[41030] INFO Stopping worker at tcp://192.168.7.20:64425
> 2019-06-18 14:24:21,379 distributed.dask_worker[41007] INFO End worker
> bash-5.0$ echo $?
> 0
>
> Would changing that to be non-zero break anything else/
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/distributed/issues/2781?email_source=notifications&email_token=AACKZTEIJX6BIJBZKKUVYF3P3EZBVA5CNFSM4HZDED3KYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4G2HIGYA>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AACKZTEMME3KP7AQJKTARV3P3EZBVANCNFSM4HZDED3A>
> .
>

OK, I'll put up a PR soon.

Resolving patch:
diff --git a/distributed/cli/dask_worker.py b/distributed/cli/dask_worker.py
--- a/distributed/cli/dask_worker.py
+++ b/distributed/cli/dask_worker.py
@@ -373,7 +373,10 @@ def run():
 
     try:
         loop.run_sync(run)
-    except (KeyboardInterrupt, TimeoutError):
+    except TimeoutError:
+        # We already log the exception in nanny / worker. Don't do it again.
+        raise TimeoutError("Timed out starting worker.") from None
+    except KeyboardInterrupt:
         pass
     finally:
         logger.info("End worker")
diff --git a/distributed/nanny.py b/distributed/nanny.py
--- a/distributed/nanny.py
+++ b/distributed/nanny.py
@@ -304,7 +304,13 @@ def instantiate(self, comm=None):
                 )
             except gen.TimeoutError:
                 yield self.close(timeout=self.death_timeout)
-                raise gen.Return("timed out")
+                logger.exception(
+                    "Timed out connecting Nanny '%s' to scheduler '%s'",
+                    self,
+                    self.scheduler_addr,
+                )
+                raise
+
         else:
             result = yield self.process.start()
         raise gen.Return(result)
diff --git a/distributed/worker.py b/distributed/worker.py
--- a/distributed/worker.py
+++ b/distributed/worker.py
@@ -708,8 +708,14 @@ def _register_with_scheduler(self):
         logger.info("-" * 49)
         while True:
             if self.death_timeout and time() > start + self.death_timeout:
+                logger.exception(
+                    "Timed out when connecting to scheduler '%s'",
+                    self.scheduler.address,
+                )
                 yield self.close(timeout=1)
-                return
+                raise gen.TimeoutError(
+                    "Timed out connecting to scheduler '%s'" % self.scheduler.address
+                )
             if self.status in ("closed", "closing"):
                 raise gen.Return
             try:


Test patch:
diff --git a/distributed/cli/tests/test_dask_worker.py b/distributed/cli/tests/test_dask_worker.py
--- a/distributed/cli/tests/test_dask_worker.py
+++ b/distributed/cli/tests/test_dask_worker.py
@@ -1,6 +1,7 @@
 from __future__ import print_function, division, absolute_import
 
 import pytest
+from click.testing import CliRunner
 
 pytest.importorskip("requests")
 
@@ -9,6 +10,7 @@
 import os
 from time import sleep
 
+import distributed.cli.dask_worker
 from distributed import Client
 from distributed.metrics import time
 from distributed.utils import sync, tmpfile
@@ -292,3 +294,15 @@ def test_dashboard_non_standard_ports(loop):
 
         with pytest.raises(Exception):
             requests.get("http://localhost:4833/status/")
+
+
+@pytest.mark.slow
+@pytest.mark.parametrize("no_nanny", [True, False])
+def test_worker_timeout(no_nanny):
+    runner = CliRunner()
+    args = ["192.168.1.100:7777", "--death-timeout=1"]
+    if no_nanny:
+        args.append("--no-nanny")
+    result = runner.invoke(distributed.cli.dask_worker.main, args)
+    assert result.exit_code != 0
+    assert str(result.exception).startswith("Timed out")
diff --git a/distributed/tests/test_nanny.py b/distributed/tests/test_nanny.py
--- a/distributed/tests/test_nanny.py
+++ b/distributed/tests/test_nanny.py
@@ -160,9 +160,10 @@ def test_nanny_alt_worker_class(c, s, w1, w2):
 @gen_cluster(client=False, nthreads=[])
 def test_nanny_death_timeout(s):
     yield s.close()
-    w = yield Nanny(s.address, death_timeout=1)
+    w = Nanny(s.address, death_timeout=1)
+    with pytest.raises(gen.TimeoutError):
+        yield w
 
-    yield gen.sleep(3)
     assert w.status == "closed"
 
 
diff --git a/distributed/tests/test_worker.py b/distributed/tests/test_worker.py
--- a/distributed/tests/test_worker.py
+++ b/distributed/tests/test_worker.py
@@ -736,9 +736,11 @@ def test_hold_onto_dependents(c, s, a, b):
 def test_worker_death_timeout(s):
     with dask.config.set({"distributed.comm.timeouts.connect": "1s"}):
         yield s.close()
-        w = yield Worker(s.address, death_timeout=1)
+        w = Worker(s.address, death_timeout=1)
+
+    with pytest.raises(gen.TimeoutError):
+        yield w
 
-    yield gen.sleep(2)
     assert w.status == "closed"