Repository: jupyterhub/jupyterhub

Issue:
Proxy entries not cleaned up when single user servers are stopped
I'm running the proxy and the hub separately in installs (this allows me to restart them independently). This leads to some issues when the hub is restarted but the proxy isn't.
1. Hub calls user.stop() on all currently running users
2. This stops all the running singleuser services, but doesn't actually remove entries from the proxy!
3. Hub comes back, correctly determins there are no singleuser servers running. But the Proxy still thinks they are, and so this state is mismatched
4. Next time when you try to spawn something, the proxy proxies it back to where the singleuser server _used_ to run, and fails with a 5xx error.
5. Restarting the proxy clears the proxy route table, bringing it back in sync with what the hub sees. This 'fixes' the issue

This doesn't manifest itself in the traditional configuration of the hub managing the proxy, since the proxy does usually get restarted with the hub. It also doesn't manifest when `c.JupyterHub.cleanup_servers = False`, since single user servers aren't stopped when the hub restarts. So it only manifests when you are running the proxy and hub separately with `cleanup_servers = True`

I think the real fix is to also cleanup the routes in the proxy when cleaning up single user servers on restart.



Discussion:
Agreed. It checks that all the routes for running containers are present, and fills them in if not, but it doesn't do the opposite - finding routes for _not_ running users, and removing those.


Resolving patch:
diff --git a/jupyterhub/app.py b/jupyterhub/app.py
--- a/jupyterhub/app.py
+++ b/jupyterhub/app.py
@@ -1112,7 +1112,7 @@ def start_proxy(self):
         if self.proxy.public_server.is_up() or self.proxy.api_server.is_up():
             # check for *authenticated* access to the proxy (auth token can change)
             try:
-                yield self.proxy.get_routes()
+                routes = yield self.proxy.get_routes()
             except (HTTPError, OSError, socket.error) as e:
                 if isinstance(e, HTTPError) and e.code == 403:
                     msg = "Did CONFIGPROXY_AUTH_TOKEN change?"
@@ -1124,6 +1124,7 @@ def start_proxy(self):
                 return
             else:
                 self.log.info("Proxy already running at: %s", self.proxy.public_server.bind_url)
+                yield self.proxy.check_routes(self.users, self._service_map, routes)
             self.proxy_process = None
             return
 
diff --git a/jupyterhub/orm.py b/jupyterhub/orm.py
--- a/jupyterhub/orm.py
+++ b/jupyterhub/orm.py
@@ -257,19 +257,17 @@ def check_routes(self, user_dict, service_dict, routes=None):
         user_routes = { r['user'] for r in routes.values() if 'user' in r }
         futures = []
         db = inspect(self).session
-        for orm_user in db.query(User).filter(User.server != None):
+        for orm_user in db.query(User):
             user = user_dict[orm_user]
-            if not user.running:
-                # Don't add users to the proxy that haven't finished starting
-                continue
-            if user.server is None:
-                # This should never be True, but seems to be on rare occasion.
-                # catch filter bug, either in sqlalchemy or my understanding of its behavior
-                self.log.error("User %s has no server, but wasn't filtered out.", user)
-                continue
-            if user.name not in user_routes:
-                self.log.warning("Adding missing route for %s (%s)", user.name, user.server)
-                futures.append(self.add_user(user))
+            if user.running:
+                if user.name not in user_routes:
+                    self.log.warning("Adding missing route for %s (%s)", user.name, user.server)
+                    futures.append(self.add_user(user))
+            else:
+                # User not running, make sure it's not in the table
+                if user.name in user_routes:
+                    self.log.warning("Removing route for not running %s", user.name)
+                    futures.append(self.delete_user(user))
         
         # check service routes
         service_routes = { r['service'] for r in routes.values() if 'service' in r }
diff --git a/jupyterhub/user.py b/jupyterhub/user.py
--- a/jupyterhub/user.py
+++ b/jupyterhub/user.py
@@ -161,9 +161,9 @@ def escaped_name(self):
     @property
     def proxy_path(self):
         if self.settings.get('subdomain_host'):
-            return url_path_join('/' + self.domain, self.server.base_url)
+            return url_path_join('/' + self.domain, self.base_url)
         else:
-            return self.server.base_url
+            return self.base_url
     
     @property
     def domain(self):


Test patch: