Repository: jupyterhub/jupyterhub

Issue:
API shutdown doesn't call cleanup
### Bug description

If a POST is sent to the shutdown API endpoint http://localhost:8000/hub/api/shutdown, either through the shutdown button on the admin page, or by making a request to the API with curl, `cleanup()`:
https://github.com/jupyterhub/jupyterhub/blob/107dc02fd0698eaff11125829a947863275f32ef/jupyterhub/app.py#L2871-L2872
is not called

#### Expected behaviour

The hub should shut down gracefully by calling `cleanup()`

#### Actual behaviour

It seems to shutdown straightaway.

### How to reproduce

Using the latest `main` Docker Hub image
```sh
podman pull jupyterhub/jupyterhub:main
podman run -it --rm -p 8000:8000 jupyterhub/jupyterhub:main bash
```
In the container run jupyterhub:
```
root@4a418a63bfba:/srv/jupyterhub# jupyterhub --JupyterHub.authenticator_class=dummy --Authenticator.admin_users=demo --debug
```
Go to http://localhost:8000/hub/admin#/ , login as `demo`, click `Shutdown Hub`. Logs:
```
[D 2022-04-30 17:14:01.292 JupyterHub scopes:792] Checking access via scope shutdown
[D 2022-04-30 17:14:01.292 JupyterHub scopes:606] Unrestricted access to /hub/api/shutdown via shutdown
[I 2022-04-30 17:14:01.293 JupyterHub log:186] 202 POST /hub/api/shutdown (demo@::ffff:10.0.2.100) 6.24ms
root@4a418a63bfba:/srv/jupyterhub#
```
`configurable-http-proxy` is still running:
```
root@4a418a63bfba:/srv/jupyterhub# ps auwx
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0   7244  3640 pts/0    Ss   17:13   0:00 bash
root           8  1.4  0.2 585092 45988 ?        Ssl  17:13   0:00 node /usr/local/bin/configurable-http-proxy --ip  --port 8000 --api-ip 127.0.0.1 --api-port 8001 --error-target http://12
root          15  0.0  0.0   8884  3232 pts/0    R+   17:14   0:00 ps auwx
```
In contrast if the hub is stopped with `<Ctrl-C>` in the terminal:
```
^C[C 2022-04-30 17:16:34.920 JupyterHub app:3239] Received signal SIGINT, initiating shutdown...
[I 2022-04-30 17:16:34.921 JupyterHub app:2883] Cleaning up single-user servers...
[D 2022-04-30 17:16:34.922 JupyterHub app:2895] Stopping proxy
[I 2022-04-30 17:16:34.922 JupyterHub proxy:750] Cleaning up proxy[8]...
[D 2022-04-30 17:16:34.923 JupyterHub proxy:607] Removing proxy pid file jupyterhub-proxy.pid
17:16:34.924 [ConfigProxy] warn: Terminated
[I 2022-04-30 17:16:34.925 JupyterHub app:2915] ...done
root@9f02d163f208:/srv/jupyterhub#
```
```
root@9f02d163f208:/srv/jupyterhub# ps auwx
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0   7244  3788 pts/0    Ss   17:15   0:00 bash
root          15  0.0  0.0   8884  3212 pts/0    R+   17:16   0:00 ps auwx
```

This is the backend part of the bug in https://github.com/jupyterhub/jupyterhub/issues/3880


Discussion:

Resolving patch:
diff --git a/jupyterhub/apihandlers/hub.py b/jupyterhub/apihandlers/hub.py
--- a/jupyterhub/apihandlers/hub.py
+++ b/jupyterhub/apihandlers/hub.py
@@ -47,9 +47,8 @@ def post(self):
         self.set_status(202)
         self.finish(json.dumps({"message": "Shutting down Hub"}))
 
-        # stop the eventloop, which will trigger cleanup
-        loop = IOLoop.current()
-        loop.add_callback(loop.stop)
+        # instruct the app to stop, which will trigger cleanup
+        app.stop()
 
 
 class RootAPIHandler(APIHandler):
diff --git a/jupyterhub/app.py b/jupyterhub/app.py
--- a/jupyterhub/app.py
+++ b/jupyterhub/app.py
@@ -3234,9 +3234,15 @@ def atexit(self):
         loop.make_current()
         loop.run_sync(self.cleanup)
 
-    async def shutdown_cancel_tasks(self, sig):
+    async def shutdown_cancel_tasks(self, sig=None):
         """Cancel all other tasks of the event loop and initiate cleanup"""
-        self.log.critical("Received signal %s, initiating shutdown...", sig.name)
+        if sig is None:
+            self.log.critical("Initiating shutdown...")
+        else:
+            self.log.critical("Received signal %s, initiating shutdown...", sig.name)
+
+        await self.cleanup()
+
         tasks = [t for t in asyncio_all_tasks() if t is not asyncio_current_task()]
 
         if tasks:
@@ -3253,7 +3259,6 @@ async def shutdown_cancel_tasks(self, sig):
             tasks = [t for t in asyncio_all_tasks()]
             for t in tasks:
                 self.log.debug("Task status: %s", t)
-        await self.cleanup()
         asyncio.get_event_loop().stop()
 
     def stop(self):
@@ -3261,7 +3266,7 @@ def stop(self):
             return
         if self.http_server:
             self.http_server.stop()
-        self.io_loop.add_callback(self.io_loop.stop)
+        self.io_loop.add_callback(self.shutdown_cancel_tasks)
 
     async def start_show_config(self):
         """Async wrapper around base start_show_config method"""


Test patch:
diff --git a/jupyterhub/tests/conftest.py b/jupyterhub/tests/conftest.py
--- a/jupyterhub/tests/conftest.py
+++ b/jupyterhub/tests/conftest.py
@@ -188,6 +188,8 @@ def cleanup_after(request, io_loop):
         if not MockHub.initialized():
             return
         app = MockHub.instance()
+        if app.db_file.closed:
+            return
         for uid, user in list(app.users.items()):
             for name, spawner in list(user.spawners.items()):
                 if spawner.active:
diff --git a/jupyterhub/tests/mocking.py b/jupyterhub/tests/mocking.py
--- a/jupyterhub/tests/mocking.py
+++ b/jupyterhub/tests/mocking.py
@@ -325,26 +325,28 @@ async def initialize(self, argv=None):
         roles.assign_default_roles(self.db, entity=user)
         self.db.commit()
 
-    def stop(self):
-        super().stop()
+    _stop_called = False
 
+    def stop(self):
+        if self._stop_called:
+            return
+        self._stop_called = True
         # run cleanup in a background thread
         # to avoid multiple eventloops in the same thread errors from asyncio
 
         def cleanup():
-            asyncio.set_event_loop(asyncio.new_event_loop())
-            loop = IOLoop.current()
-            loop.run_sync(self.cleanup)
+            loop = asyncio.new_event_loop()
+            loop.run_until_complete(self.cleanup())
             loop.close()
 
-        pool = ThreadPoolExecutor(1)
-        f = pool.submit(cleanup)
-        # wait for cleanup to finish
-        f.result()
-        pool.shutdown()
+        with ThreadPoolExecutor(1) as pool:
+            f = pool.submit(cleanup)
+            # wait for cleanup to finish
+            f.result()
 
-        # ignore the call that will fire in atexit
-        self.cleanup = lambda: None
+        # prevent redundant atexit from running
+        self._atexit_ran = True
+        super().stop()
         self.db_file.close()
 
     async def login_user(self, name):
diff --git a/jupyterhub/tests/test_api.py b/jupyterhub/tests/test_api.py
--- a/jupyterhub/tests/test_api.py
+++ b/jupyterhub/tests/test_api.py
@@ -2095,14 +2095,23 @@ async def shutdown():
         )
         return r
 
-    real_stop = loop.stop
+    real_stop = loop.asyncio_loop.stop
 
     def stop():
         stop.called = True
         loop.call_later(1, real_stop)
 
-    with mock.patch.object(loop, 'stop', stop):
+    real_cleanup = app.cleanup
+
+    def cleanup():
+        cleanup.called = True
+        return real_cleanup()
+
+    app.cleanup = cleanup
+
+    with mock.patch.object(loop.asyncio_loop, 'stop', stop):
         r = loop.run_sync(shutdown, timeout=5)
     r.raise_for_status()
     reply = r.json()
+    assert cleanup.called
     assert stop.called