Repository: jupyterhub/jupyterhub

Issue:
Parallelize user server status check
At startup, the hub takes a long time to check the status of user servers when there are thousands of accounts.


Discussion:
This takes several minutes for us, with 5k accounts.

On Thu, Jan 25, 2018 at 2:41 PM, Ryan Lovett <notifications@github.com>
wrote:

> At startup, the hub takes a long time to check the status of user servers
> when there are thousands of accounts.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/jupyterhub/jupyterhub/issues/1633>, or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAB23uQAa9JhFLeLG3-jEdJ53owKMtHIks5tOQMFgaJpZM4Rtmbq>
> .
>



-- 
Yuvi Panda T
http://yuvi.in/blog

At startup, [`check_hub_version`](https://github.com/jupyterhub/jupyterhub/blob/c9db504a49deb57d3494ce331fc6c904cf0a9e57/jupyterhub/singleuser.py#L341) is called within `start`:
```
   def start(self):
        self.log.info("Starting jupyterhub-singleuser server version %s", __version__)
        # start by hitting Hub to check version
        ioloop.IOLoop.current().run_sync(self.check_hub_version)
        super(SingleUserNotebookApp, self).start()
```
- How important is checking the version now?
Thanks for taking a look, @willingc!

I think that code is in the singleuser server, which is not where we discovered this problem. This is in the core hub itself.

When it starts up, it tries to poll every single user (https://github.com/jupyterhub/jupyterhub/blob/master/jupyterhub/app.py#L1589) and make sure they have routes in the proxy. This takes a *long time* when you have 5000 users, and during this time the hub is unavailable to serve requests, although I'm not sure why.

Hmm...

- app.py https://github.com/jupyterhub/jupyterhub/blob/master/jupyterhub/app.py#L1589
- proxy.py https://github.com/jupyterhub/jupyterhub/blob/master/jupyterhub/proxy.py#L291

This caused an outage again just now, since it took more than 20mins to come up :(
We fixed it temporarily by deleting the user accounts of every user not logged in. This shouldn't affect their homedirs in any way.

I ran:

```sql
delete from users where users.id in (select users.id from users join spawners on spawners.user_id = users.id where server_id is null);
```

To do this.
The logs slowly fill with messages like:

```
[I 2018-01-26 23:51:51.273 JupyterHub app:1223] <user> still running
```

over and over again, one for each user running.


After a bunch more digging, it looks like the code that needs parallelizing is https://github.com/jupyterhub/jupyterhub/blob/master/jupyterhub/app.py#L1205
The core reason for this being *super* slow turned out to be https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/461, which was spawning too many threads and thrashing things about. But this should still be parallelized tho!

Resolving patch:
diff --git a/jupyterhub/app.py b/jupyterhub/app.py
--- a/jupyterhub/app.py
+++ b/jupyterhub/app.py
@@ -1204,8 +1204,6 @@ def check_services_health(self):
     def init_spawners(self):
         db = self.db
 
-        user_summaries = ['']
-
         def _user_summary(user):
             parts = ['{0: >8}'.format(user.name)]
             if user.admin:
@@ -1225,39 +1223,49 @@ def user_stopped(user, server_name):
             yield self.proxy.delete_user(user, server_name)
             yield user.stop(server_name)
 
+        @gen.coroutine
+        def check_spawner(user, name, spawner):
+            status = 0
+            if spawner.server:
+                try:
+                    status = yield spawner.poll()
+                except Exception:
+                    self.log.exception("Failed to poll spawner for %s, assuming the spawner is not running.",
+                        spawner._log_name)
+                    status = -1
+
+            if status is None:
+                self.log.info("%s still running", user.name)
+                spawner.add_poll_callback(user_stopped, user, name)
+                spawner.start_polling()
+            else:
+                # user not running. This is expected if server is None,
+                # but indicates the user's server died while the Hub wasn't running
+                # if spawner.server is defined.
+                if spawner.server:
+                    self.log.warning("%s appears to have stopped while the Hub was down", spawner._log_name)
+                    # remove server entry from db
+                    db.delete(spawner.orm_spawner.server)
+                    spawner.server = None
+                else:
+                    self.log.debug("%s not running", spawner._log_name)
+
+        # parallelize checks for running Spawners
+        check_futures = []
         for orm_user in db.query(orm.User):
             self.users[orm_user.id] = user = User(orm_user, self.tornado_settings)
             self.log.debug("Loading state for %s from db", user.name)
             for name, spawner in user.spawners.items():
-                status = 0
-                if spawner.server:
-                    try:
-                        status = yield spawner.poll()
-                    except Exception:
-                        self.log.exception("Failed to poll spawner for %s, assuming the spawner is not running.",
-                            spawner._log_name)
-                        status = -1
-
-                if status is None:
-                    self.log.info("%s still running", user.name)
-                    spawner.add_poll_callback(user_stopped, user, name)
-                    spawner.start_polling()
-                else:
-                    # user not running. This is expected if server is None,
-                    # but indicates the user's server died while the Hub wasn't running
-                    # if spawner.server is defined.
-                    if spawner.server:
-                        self.log.warning("%s appears to have stopped while the Hub was down", spawner._log_name)
-                        # remove server entry from db
-                        db.delete(spawner.orm_spawner.server)
-                        spawner.server = None
-                    else:
-                        self.log.debug("%s not running", spawner._log_name)
-            db.commit()
+                f = check_spawner(user, name, spawner)
+                check_futures.append(f)
+
+        # await checks after submitting them all
+        for f in check_futures:
+            yield f
 
-            user_summaries.append(_user_summary(user))
+        user_summaries = map(_user_summary, self.users.values())
 
-        self.log.debug("Loaded users: %s", '\n'.join(user_summaries))
+        self.log.debug("Loaded users:\n%s", '\n'.join(user_summaries))
         db.commit()
 
     def init_oauth(self):


Test patch: