Repository: dask/distributed

Issue:
Task pending message on client close
I noticed on the current `main` branch I've started seeing a `Task was destroyed but it is pending!` message printed when I exit an ipython session. Looking through recent commits, it appears 867a54e7e71d6f16449f3119f015c69ecf8d5b3a may be where this message started appearing. 

Locally when I check out 4e2c1b36a9cfb7df092f47ca604511319cbeab52 (the commit just prior to 867a54e7e71d6f16449f3119f015c69ecf8d5b3a) I don't see this message:

```python
(distributed) ➜  distributed git:(4e2c1b36) ipython
Python 3.9.7 | packaged by conda-forge | (default, Sep 14 2021, 01:18:03)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.27.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from distributed import Client

In [2]: client = Client()

In [3]: exit
```

and with 867a54e7e71d6f16449f3119f015c69ecf8d5b3a, we do see this message:

```python
(distributed) ➜  distributed git:(867a54e7) ipython
Python 3.9.7 | packaged by conda-forge | (default, Sep 14 2021, 01:18:03)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.27.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from distributed import Client

In [2]: client = Client()

In [3]: exit
Task was destroyed but it is pending!
task: <Task pending name='Task-184' coro=<Client._handle_report() running at /Users/james/projects/dask/distributed/distributed/client.py:1360> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x1213ff130>()]>>
```

cc @graingert 


Discussion:

Resolving patch:
diff --git a/distributed/client.py b/distributed/client.py
--- a/distributed/client.py
+++ b/distributed/client.py
@@ -1515,12 +1515,11 @@ async def _close(self, fast=False):
                 _set_global_client(None)
 
             if (
-                not fast
-                and handle_report_task is not None
+                handle_report_task is not None
                 and handle_report_task is not current_task
             ):
                 with suppress(TimeoutError, asyncio.CancelledError):
-                    await asyncio.wait_for(handle_report_task, 2)
+                    await asyncio.wait_for(handle_report_task, 0 if fast else 2)
 
             with suppress(AttributeError):
                 await self.scheduler.close_rpc()


Test patch:
diff --git a/distributed/tests/test_client.py b/distributed/tests/test_client.py
--- a/distributed/tests/test_client.py
+++ b/distributed/tests/test_client.py
@@ -7139,6 +7139,28 @@ async def async_user_event_handler(event):
     assert len(all_events) == 3
 
 
+@gen_cluster(client=True, nthreads=[("", 1)])
+async def test_events_subscribe_topic_cancelled(c, s, a):
+    event_handler_started = asyncio.Event()
+    exc_info = None
+
+    async def user_event_handler(event):
+        nonlocal exc_info
+        c.unsubscribe_topic("test-topic")
+        event_handler_started.set()
+        with pytest.raises(asyncio.CancelledError) as exc_info:
+            await asyncio.sleep(0.5)
+
+    c.subscribe_topic("test-topic", user_event_handler)
+    while not s.event_subscriber["test-topic"]:
+        await asyncio.sleep(0.01)
+
+    a.log_event("test-topic", {})
+    await event_handler_started.wait()
+    await c._close(fast=True)
+    assert exc_info is not None
+
+
 @gen_cluster(client=True, nthreads=[("", 1)])
 async def test_events_all_servers_use_same_channel(c, s, a):
     """Ensure that logs from all server types (scheduler, worker, nanny)