Repository: dask/distributed

Issue:
Scheduler timeout fails to exit `dask-scheduler` process
The scheduler timeout introduced in #2652 seems to successfully close the scheduler comms, however the process doesn't exit.

I would expect the process to also exit when the scheduler has closed.

```console
$ pip install git+https://github.com/dask/distributed.git

$ export DASK_DISTRIBUTED__SCHEDULER__IDLE_TIMEOUT='200 ms'

$ python -c 'import distributed; print(distributed.__version__)' 
2.1.0+4.ge5ec8daa 

$ dask-scheduler  
distributed.scheduler - INFO - ----------------------------------------------- 
distributed.scheduler - INFO - Clear task state 
distributed.dashboard.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: pip install jupyter-server-proxy
distributed.scheduler - INFO -   Scheduler at:     tcp://172.17.0.2:8786 
distributed.scheduler - INFO -   dashboard at:                     :8787
distributed.scheduler - INFO - Local Directory:    /tmp/scheduler-gx0ramly
distributed.scheduler - INFO - -----------------------------------------------
distributed.scheduler - INFO - Scheduler closing... 
distributed.scheduler - INFO - Scheduler closing all comms


... (hit enter a bunch of times and waited a while here before hitting CTRL+c)...


^Cdistributed.scheduler - INFO - End scheduler at 'tcp://172.17.0.2:8786' 
```

The scheduler should exit cleanly to satisfy the requirements in #2346.


Discussion:

Resolving patch:
diff --git a/distributed/cli/dask_scheduler.py b/distributed/cli/dask_scheduler.py
--- a/distributed/cli/dask_scheduler.py
+++ b/distributed/cli/dask_scheduler.py
@@ -120,6 +120,12 @@
 @click.argument(
     "preload_argv", nargs=-1, type=click.UNPROCESSED, callback=validate_preload_argv
 )
+@click.option(
+    "--idle-timeout",
+    default=None,
+    type=str,
+    help="Time of inactivity after which to kill the scheduler",
+)
 @click.version_option()
 def main(
     host,
@@ -141,6 +147,7 @@ def main(
     tls_cert,
     tls_key,
     dashboard_address,
+    idle_timeout,
 ):
     g0, g1, g2 = gc.get_threshold()  # https://github.com/dask/distributed/issues/1653
     gc.set_threshold(g0 * 3, g1 * 3, g2 * 3)
@@ -211,6 +218,7 @@ def del_pid_file():
         protocol=protocol,
         dashboard_address=dashboard_address if dashboard else None,
         service_kwargs={"dashboard": {"prefix": dashboard_prefix}},
+        idle_timeout=idle_timeout,
     )
     scheduler.start()
     if not preload:
@@ -226,9 +234,12 @@ def del_pid_file():
 
     install_signal_handlers(loop)
 
+    async def run():
+        await scheduler
+        await scheduler.finished()
+
     try:
-        loop.start()
-        loop.close()
+        loop.run_sync(run)
     finally:
         scheduler.stop()
         if local_directory_created:


Test patch:
diff --git a/distributed/cli/tests/test_dask_scheduler.py b/distributed/cli/tests/test_dask_scheduler.py
--- a/distributed/cli/tests/test_dask_scheduler.py
+++ b/distributed/cli/tests/test_dask_scheduler.py
@@ -383,3 +383,14 @@ def test_version_option():
     runner = CliRunner()
     result = runner.invoke(distributed.cli.dask_scheduler.main, ["--version"])
     assert result.exit_code == 0
+
+
+@pytest.mark.slow
+def test_idle_timeout(loop):
+    start = time()
+    runner = CliRunner()
+    result = runner.invoke(
+        distributed.cli.dask_scheduler.main, ["--idle-timeout", "1s"]
+    )
+    stop = time()
+    assert 1 < stop - start < 10