Repository: jupyterhub/jupyterhub
Pull request: Fix asyncio deprecation asyncio.Task.all_tasks
This is to fix #3297 

Problem statement:
Cannot shutdown on python 3.9 due to asyncio deprecation
### Bug description
When using jupyterhub with python3.9, you will not be able to shutdown (e.g. via ctrl+c). When attempting to stop asyncio tasks, you will get `type object '_asyncio.Task' has no attribute 'all_tasks'` error. `asyncio.Task.all_tasks()` and `current_task()` have been deprecated since python 3.7 and is fully removed in 3.9.  So all references to `asyncio.Task.all_tasks()` should change to `asyncio.all_tasks()`.

I have a fix that will accommodate python3.6 and later. It's quite a simple fix. I hope it fits the conventions that this project uses. btw, if support for python3.6 can be dropped, then the change is even simpler and without any clutter.

#### Expected behaviour
jupyterhub should terminate gracefully like:
```
^C[C 2020-12-02 17:50:48.625 JupyterHub app:2826] Received signal SIGINT, initiating shutdown...
[I 2020-12-02 17:50:48.626 JupyterHub app:2467] Cleaning up single-user servers...
[I 2020-12-02 17:50:48.627 JupyterHub proxy:735] Cleaning up proxy[9371]...
17:50:48.633 [ConfigProxy] warn: Terminated
[I 2020-12-02 17:50:48.638 JupyterHub app:2499] ...done
```

#### Actual behaviour
```
^C[C 2020-12-02 16:16:45.801 JupyterHub app:2810] Received signal SIGINT, initiating shutdown...
ERROR:asyncio:Task exception was never retrieved
future: <Task finished name='Task-8' coro=<JupyterHub.shutdown_cancel_tasks() done, defined at /opt/miniconda3/envs/jupyterhub/lib/python3.9/site-packages/jupyterhub/app.py:2808> exception=AttributeError("type object '_asyncio.Task' has no attribute 'all_tasks'")>
Traceback (most recent call last):
  File "/opt/miniconda3/envs/jupyterhub/lib/python3.9/site-packages/jupyterhub/app.py", line 2812, in shutdown_cancel_tasks
    t for t in asyncio.Task.all_tasks() if t is not asyncio.Task.current_task()
AttributeError: type object '_asyncio.Task' has no attribute 'all_tasks'
```

### How to reproduce
I'm using conda to install python and jupyterhub
```
conda create -n jupyterhub python=3.9 jupyterhub
jupyterhub
ctrl-c
```

### Your personal set up
 - OS:
Ubuntu  20.04.1
 - Version:
jupyterhub 1.2.2
 - Configuration:
using conda, as shown above.

Hints from issue discussion:

Commit message: Fix asyncio deprecation asyncio.Task.all_tasks

Patch to review:
diff --git a/jupyterhub/app.py b/jupyterhub/app.py
index b6d8571ad9..6cf4b093a2 100644
--- a/jupyterhub/app.py
+++ b/jupyterhub/app.py
@@ -29,6 +29,13 @@
 if sys.version_info[:2] < (3, 3):
     raise ValueError("Python < 3.3 not supported: %s" % sys.version)
 
+# For compatibility with python versions 3.6 or earlier.
+# asyncio.Task.all_tasks() is fully moved to asyncio.all_tasks() starting with 3.9. Also applies to current_task.
+try:
+    asyncio.all_tasks
+except AttributeError as e:
+    asyncio.all_tasks = asyncio.Task.all_tasks
+    asyncio.current_task = asyncio.Task.current_task
 
 from dateutil.parser import parse as parse_date
 from jinja2 import Environment, FileSystemLoader, PrefixLoader, ChoiceLoader
@@ -2817,9 +2824,7 @@ def atexit(self):
     async def shutdown_cancel_tasks(self, sig):
         """Cancel all other tasks of the event loop and initiate cleanup"""
         self.log.critical("Received signal %s, initiating shutdown...", sig.name)
-        tasks = [
-            t for t in asyncio.Task.all_tasks() if t is not asyncio.Task.current_task()
-        ]
+        tasks = [t for t in asyncio.all_tasks() if t is not asyncio.current_task()]
 
         if tasks:
             self.log.debug("Cancelling pending tasks")
@@ -2832,7 +2837,7 @@ async def shutdown_cancel_tasks(self, sig):
             except StopAsyncIteration as e:
                 self.log.error("Caught StopAsyncIteration Exception", exc_info=True)
 
-            tasks = [t for t in asyncio.Task.all_tasks()]
+            tasks = [t for t in asyncio.all_tasks()]
             for t in tasks:
                 self.log.debug("Task status: %s", t)
         await self.cleanup()
diff --git a/jupyterhub/utils.py b/jupyterhub/utils.py
index 8fcf7bc6a6..a5b221e82d 100644
--- a/jupyterhub/utils.py
+++ b/jupyterhub/utils.py
@@ -28,6 +28,14 @@
 from tornado.log import app_log
 from tornado.platform.asyncio import to_asyncio_future
 
+# For compatibility with python versions 3.6 or earlier.
+# asyncio.Task.all_tasks() is fully moved to asyncio.all_tasks() starting with 3.9. Also applies to current_task.
+try:
+    asyncio.all_tasks
+except AttributeError as e:
+    asyncio.all_tasks = asyncio.Task.all_tasks
+    asyncio.current_task = asyncio.Task.current_task
+
 
 def random_port():
     """Get a single random port."""
@@ -474,7 +482,7 @@ def print_stacks(file=sys.stderr):
     # also show asyncio tasks, if any
     # this will increase over time as we transition from tornado
     # coroutines to native `async def`
-    tasks = asyncio.Task.all_tasks()
+    tasks = asyncio.all_tasks()
     if tasks:
         print("AsyncIO tasks: %i" % len(tasks))
         for task in tasks:


Code review comments:
diff_hunk: @@ -29,6 +29,13 @@
 if sys.version_info[:2] < (3, 3):
     raise ValueError("Python < 3.3 not supported: %s" % sys.version)
 
+# For compatibility with python versions 3.6 or earlier.
+# asyncio.Task.all_tasks() is fully moved to asyncio.all_tasks() starting with 3.9. Also applies to current_task.
+try:
+    asyncio.all_tasks
+except AttributeError as e:
+    asyncio.all_tasks = asyncio.Task.all_tasks
path: jupyterhub/app.py
text: @user1:
This is monkeypatching the asyncio Python module in place. It's probably fine, though monkeypatching can occasionally lead to some weird corner cases or confuse people debugging problems. What do you think about assigning it to a variable instead, e.g.
```py
try:
    asyncio_all_tasks = asyncio.all_tasks
except AttributeError as e:
    asyncio_all_tasks = asyncio.Task.all_tasks
    ...
```

@author:
Ah, I monkypatched the asyncio module so as to make it easier to remove in future, if python3.6 support can be dropped. Of course, making it a variable will also work, and then when it's time to move away from python3.6 we can replace the variable usage instead.

I'm not sure if you think this convenience is worth monkeypatching it though?

@user2:
I like the PR's compromise over this suggested compromise unless we actually override something in asyncio but only adds an entry, which I assume we don't.

I mostly consider this from a readability standpoint, and have little experience with issues caused by modifying classes.

@user3:
Even though I think it's highly unlikely to cause an issue, I think we should definitely go with @user1 suggestion. Assigning attributes on a stdlib module isn't something i think we should be doing. 

@author:
Ah ok, I've changed the monkeypatching to using variables instead, so we don't modify anything on modules.