Repository: dask/distributed

Issue:
jupyter_client 7 breaks CI
After jupyter_client got bumped from 6.1.12 to 7.0.1 upstream, CI started failing:

All Python versions on Linux and MacOS:
- FAILED distributed/tests/test_utils.py::test_loop_runner
- FAILED distributed/tests/test_utils.py::test_two_loop_runners

Python 3.7 and 3.8 on Windows:
- FAILED distributed/tests/test_client.py::test_get_client_functions_spawn_clusters
- FAILED distributed/tests/test_worker.py::test_worker_client_closes_if_created_on_worker_one_worker
- FAILED distributed/tests/test_worker.py::test_worker_client_closes_if_created_on_worker_last_worker_alive
- FAILED distributed/tests/test_ipython.py::test_start_ipython_workers
- FAILED distributed/tests/test_ipython.py::test_start_ipython_scheduler
- FAILED distributed/tests/test_ipython.py::test_start_ipython_scheduler_magic
- FAILED distributed/tests/test_ipython.py::test_start_ipython_workers_magic
- FAILED distributed/tests/test_ipython.py::test_start_ipython_workers_magic_asterix
- FAILED distributed/tests/test_ipython.py::test_start_ipython_remote
- FAILED distributed/tests/test_ipython.py::test_start_ipython_qtconsole
- FAILED distributed/tests/test_semaphore.py::test_release_simple
- FAILED distributed/tests/test_semaphore.py::test_access_semaphore_by_name
- FAILED distributed/tests/test_semaphore.py::test_close_async
- FAILED distributed/tests/test_semaphore.py::test_release_once_too_many_resilience
- FAILED distributed/tests/test_semaphore.py::test_oversubscribing_leases
- FAILED distributed/tests/test_semaphore.py::test_threadpoolworkers_pick_correct_ioloop

Python 3.9 on Windows is, weirdly, unaffected.


Discussion:
Thanks for recording all this @crusaderky. Perhaps @minrk might have some thoughts on what could be going on? 
Looking into it. Judging by the failures, my guess is it's related to asyncio changes and how those might interact with threads.

In particular, [this change](https://github.com/jupyter/jupyter_client/compare/6.1.12...v7.0.1#diff-25e50cab312aff2f8a21c30f3ae6fd351e0adf989bcfa67d9219d70ddea05a4aR11-R12) sets the asyncio event loop policy at import time, which is not really appropriate. I'll submit a proposal to revert that. That should only affect Windows, though. My guess is a policy override is resulting in no default event loop in background threads, which is causing most of the windows failures.
I've tracked down the non-Windows issue to the new [run_sync](https://github.com/jupyter/jupyter_client/compare/6.1.12...v7.0.1#diff-25e50cab312aff2f8a21c30f3ae6fd351e0adf989bcfa67d9219d70ddea05a4aR15-R28) wrapper which uses nest_asyncio to call `run_until_complete` on an already running loop. This seems to violate some assumption in the runner test, though I don't quite understand how yet.

The failure can be reproduced by running one of the IPython tests prior to the loop runner test:

```
pytest -vsx distributed/tests/test_ipython.py::test_start_ipython_scheduler distributed/tests/test_utils.py::test_loop_runner
```

which produces the failure, due to some apparently lasting side effects on the existing loop from the `nest_asyncio` call:

<details>
<summary>pytest output</summary>

```
===================================================================================================================================================================================================== test session starts ======================================================================================================================================================================================================
platform darwin -- Python 3.8.10, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /Users/minrk/conda/envs/dask-distributed/bin/python
cachedir: .pytest_cache
rootdir: /Users/minrk/dev/py/distributed, configfile: setup.cfg
plugins: repeat-0.8.0, asyncio-0.12.0, rerunfailures-10.1, timeout-1.4.2
timeout: 300.0s
timeout method: thread
timeout func_only: False
collected 2 items

distributed/tests/test_ipython.py::test_start_ipython_scheduler distributed.http.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy
distributed.scheduler - INFO - Clear task state
distributed.scheduler - INFO -   Scheduler at:     tcp://127.0.0.1:50155
distributed.scheduler - INFO -   dashboard at:            127.0.0.1:8787
distributed.worker - INFO -       Start worker at:      tcp://127.0.0.1:50157
distributed.worker - INFO -          Listening to:      tcp://127.0.0.1:50157
distributed.worker - INFO -          dashboard at:            127.0.0.1:50158
distributed.worker - INFO - Waiting to connect to:      tcp://127.0.0.1:50155
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -                Memory:                  16.00 GiB
distributed.worker - INFO -       Local Directory: /Users/minrk/dev/py/distributed/_test_worker-2c702518-7d09-4f6c-990d-470d765ad992/dask-worker-space/worker-6rbbfa8d
distributed.worker - INFO - -------------------------------------------------
distributed.scheduler - INFO - Register worker <WorkerState 'tcp://127.0.0.1:50157', name: tcp://127.0.0.1:50157, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:50157
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to:      tcp://127.0.0.1:50155
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Receive client connection: Client-f0e016ae-0963-11ec-8414-784f4385c030
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Starting the kernel at pid: 33820
distributed.scheduler - INFO - Loading IPython extension: storemagic
reusing loop
distributed.scheduler - INFO - Remove client Client-f0e016ae-0963-11ec-8414-784f4385c030
distributed.scheduler - INFO - Remove client Client-f0e016ae-0963-11ec-8414-784f4385c030
distributed.scheduler - INFO - Close client connection: Client-f0e016ae-0963-11ec-8414-784f4385c030
distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:50157
PASSED
distributed/tests/test_utils.py::test_loop_runner FAILED

=========================================================================================================================================================================================================== FAILURES ===========================================================================================================================================================================================================
_______________________________________________________________________________________________________________________________________________________________________________________________________ test_loop_runner _______________________________________________________________________________________________________________________________________________________________________________________________________

loop_in_thread = <tornado.platform.asyncio.AsyncIOLoop object at 0x7fda95bd24c0>

    def test_loop_runner(loop_in_thread):

        # Implicit loop
        loop = IOLoop()
        loop.make_current()
        runner = LoopRunner()
        assert runner.loop not in (loop, loop_in_thread)
        assert not runner.is_started()
        assert_not_running(runner.loop)
        runner.start()
        assert runner.is_started()
        assert_running(runner.loop)
        runner.stop()
        assert not runner.is_started()
        assert_not_running(runner.loop)

        # Explicit loop
        loop = IOLoop()
        runner = LoopRunner(loop=loop)
        assert runner.loop is loop
        assert not runner.is_started()
        assert_not_running(loop)
        runner.start()
        assert runner.is_started()
        assert_running(loop)
        runner.stop()
        assert not runner.is_started()
        assert_not_running(loop)

        # Explicit loop, already started
        runner = LoopRunner(loop=loop_in_thread)
        assert not runner.is_started()
        assert_running(loop_in_thread)
>       runner.start()

distributed/tests/test_utils.py:355:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
distributed/utils.py:377: in start
    self._start_unlocked()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <distributed.utils.LoopRunner object at 0x7fda95c13820>

    def _start_unlocked(self):
        assert not self._started

        count, real_runner = self._all_loops[self._loop]
        if self._asynchronous or real_runner is not None or count > 0:
            self._all_loops[self._loop] = count + 1, real_runner
            self._started = True
            return

        assert self._loop_thread is None
        assert count == 0

        loop_evt = threading.Event()
        done_evt = threading.Event()
        in_thread = [None]
        start_exc = [None]

        def loop_cb():
            in_thread[0] = threading.current_thread()
            loop_evt.set()

        def run_loop(loop=self._loop):
            loop.add_callback(loop_cb)
            try:
                loop.start()
            except Exception as e:
                start_exc[0] = e
            finally:
                done_evt.set()

        thread = threading.Thread(target=run_loop, name="IO loop")
        thread.daemon = True
        thread.start()

        loop_evt.wait(timeout=10)
        self._started = True

        actual_thread = in_thread[0]
        if actual_thread is not thread:
            # Loop already running in other thread (user-launched)
            done_evt.wait(5)
            if not isinstance(start_exc[0], RuntimeError):
                if not isinstance(
                    start_exc[0], Exception
                ):  # track down infrequent error
>                   raise TypeError("not an exception", start_exc[0])
E                   TypeError: ('not an exception', None)

distributed/utils.py:424: TypeError
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Captured log call -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ERROR    asyncio:base_events.py:1707 Exception in callback BaseSelectorEventLoop._read_from_self()
handle: <Handle BaseSelectorEventLoop._read_from_self()>
Traceback (most recent call last):
  File "/Users/minrk/conda/envs/dask-distributed/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
RuntimeError: cannot enter context: <Context object at 0x7fda95bdbc40> is already entered
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Captured log teardown -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ERROR    asyncio:base_events.py:1707 Exception in callback BaseSelectorEventLoop._read_from_self()
handle: <Handle BaseSelectorEventLoop._read_from_self()>
Traceback (most recent call last):
  File "/Users/minrk/conda/envs/dask-distributed/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
RuntimeError: cannot enter context: <Context object at 0x7fda95bdbc40> is already entered
===================================================================================================================================================================================================== slowest 20 durations =====================================================================================================================================================================================================
5.10s call     distributed/tests/test_utils.py::test_loop_runner
2.50s call     distributed/tests/test_ipython.py::test_start_ipython_scheduler
0.00s setup    distributed/tests/test_utils.py::test_loop_runner
0.00s setup    distributed/tests/test_ipython.py::test_start_ipython_scheduler
0.00s teardown distributed/tests/test_ipython.py::test_start_ipython_scheduler
0.00s teardown distributed/tests/test_utils.py::test_loop_runner
=================================================================================================================================================================================================== short test summary info ====================================================================================================================================================================================================
FAILED distributed/tests/test_utils.py::test_loop_runner - TypeError: ('not an exception', None)
```
</details>

In fact, it can be reproduced by adding:

```python
    import nest_asyncio
    nest_asyncio.apply(asyncio.get_event_loop())
```

to the top of test_loop_runner, and can be reproduced running just that test.


Resolving patch:
diff --git a/distributed/_ipython_utils.py b/distributed/_ipython_utils.py
--- a/distributed/_ipython_utils.py
+++ b/distributed/_ipython_utils.py
@@ -76,10 +76,8 @@ def register_worker_magic(connection_info, magic_name="worker"):
     which run the given cell in a remote kernel.
     """
     ip = get_ipython()
-    info = dict(connection_info)  # copy
-    key = info.pop("key")
-    kc = BlockingKernelClient(**connection_info)
-    kc.session.key = key
+    kc = BlockingKernelClient()
+    kc.load_connection_info(connection_info)
     kc.start_channels()
 
     def remote(line, cell=None):
@@ -122,13 +120,12 @@ def remote_magic(line, cell=None):
 
     # turn info dict to hashable str for use as lookup key in _clients cache
     key = ",".join(map(str, sorted(connection_info.items())))
-    session_key = connection_info.pop("key")
 
     if key in remote_magic._clients:
         kc = remote_magic._clients[key]
     else:
-        kc = BlockingKernelClient(**connection_info)
-        kc.session.key = session_key
+        kc = BlockingKernelClient()
+        kc.load_connection_info(connection_info)
         kc.start_channels()
         kc.wait_for_ready(timeout=10)
         remote_magic._clients[key] = kc
diff --git a/distributed/utils.py b/distributed/utils.py
--- a/distributed/utils.py
+++ b/distributed/utils.py
@@ -399,8 +399,13 @@ def loop_cb():
 
         def run_loop(loop=self._loop):
             loop.add_callback(loop_cb)
+            # run loop forever if it's not running already
             try:
-                loop.start()
+                if (
+                    getattr(loop, "asyncio_loop", None) is None
+                    or not loop.asyncio_loop.is_running()
+                ):
+                    loop.start()
             except Exception as e:
                 start_exc[0] = e
             finally:
@@ -417,11 +422,13 @@ def run_loop(loop=self._loop):
         if actual_thread is not thread:
             # Loop already running in other thread (user-launched)
             done_evt.wait(5)
-            if not isinstance(start_exc[0], RuntimeError):
+            if start_exc[0] is not None and not isinstance(start_exc[0], RuntimeError):
                 if not isinstance(
                     start_exc[0], Exception
                 ):  # track down infrequent error
-                    raise TypeError("not an exception", start_exc[0])
+                    raise TypeError(
+                        f"not an exception: {start_exc[0]!r}",
+                    )
                 raise start_exc[0]
             self._all_loops[self._loop] = count + 1, None
         else:


Test patch:
diff --git a/distributed/tests/test_ipython.py b/distributed/tests/test_ipython.py
--- a/distributed/tests/test_ipython.py
+++ b/distributed/tests/test_ipython.py
@@ -25,9 +25,8 @@ def test_start_ipython_workers(loop, zmq_ctx):
         with Client(s["address"], loop=loop) as e:
             info_dict = e.start_ipython_workers()
             info = first(info_dict.values())
-            key = info.pop("key")
-            kc = BlockingKernelClient(**info)
-            kc.session.key = key
+            kc = BlockingKernelClient()
+            kc.load_connection_info(info)
             kc.start_channels()
             kc.wait_for_ready(timeout=10)
             msg_id = kc.execute("worker")
@@ -45,9 +44,8 @@ def test_start_ipython_scheduler(loop, zmq_ctx):
     with cluster(1) as (s, [a]):
         with Client(s["address"], loop=loop) as e:
             info = e.start_ipython_scheduler()
-            key = info.pop("key")
-            kc = BlockingKernelClient(**info)
-            kc.session.key = key
+            kc = BlockingKernelClient()
+            kc.load_connection_info(info)
             kc.start_channels()
             msg_id = kc.execute("scheduler")
             reply = kc.get_shell_msg(timeout=10)