Repository: jupyterhub/jupyterhub

Issue:
Surprising behavior when no state is kept in spawner
In https://github.com/jupyterhub/jupyterhub/blob/master/jupyterhub/app.py#L1061, we assume that if a user's spawner has no saved state, the server doesn't exist. This isn't necessarily the case for all spawners - the kubernetes spawner right now doesn't actually need to store any state (since it can derive all the state it needs from configuration + username), and this leads to super confusing behavior. Restarting the Hub doesn't make the hub notice the still running servers, but since kubernetes can't have two pods with same name, spawning a new container fails.

I don't know the reason we assume empty state means no server - where is that true? It's not necessarily true everywhere, so we should either clearly document that or figure out a clearer way of accomplishing what that was trying to accomplish.



Discussion:
> I don't know the reason we assume empty state means no server - where is that true?

In every Spawner that exists so far, I think. I can remove this, but I'll need to be careful, because it puts existing Spawners into a new undefined condition. As it stands, `poll` will never be called if a Spawner has been through neither `load_state` nor `start`. Removing the assumption that Spawners have state means that `poll` must be able to be the first method called, before the first call to start. This would be a new API requirement of Spawners, and I'll have to investigate how careful I need to be when introducing that. On first glance, LocalProcessSpawner is fine and handles this okay already.


Resolving patch:
diff --git a/jupyterhub/app.py b/jupyterhub/app.py
--- a/jupyterhub/app.py
+++ b/jupyterhub/app.py
@@ -1058,11 +1058,6 @@ def user_stopped(user):
 
         for orm_user in db.query(orm.User):
             self.users[orm_user.id] = user = User(orm_user, self.tornado_settings)
-            if not user.state:
-                # without spawner state, server isn't valid
-                user.server = None
-                user_summaries.append(_user_summary(user))
-                continue
             self.log.debug("Loading state for %s from db", user.name)
             spawner = user.spawner
             status = yield spawner.poll()
diff --git a/jupyterhub/spawner.py b/jupyterhub/spawner.py
--- a/jupyterhub/spawner.py
+++ b/jupyterhub/spawner.py
@@ -191,7 +191,7 @@ def __init__(self, **kwargs):
     def load_state(self, state):
         """load state from the database
         
-        This is the extensible part of state
+        This is the extensible part of state.
         
         Override in a subclass if there is state to load.
         Should call `super`.
@@ -350,7 +350,27 @@ def stop(self, now=False):
     def poll(self):
         """Check if the single-user process is running
 
-        return None if it is, an exit status (0 if unknown) if it is not.
+        returns:
+        
+        None, if single-user process is running.
+        Exit status (0 if unknown), if it is not running.
+
+        State transitions, behavior, and return response:
+
+        - If the Spawner has not been initialized (neither loaded state, nor called start),
+          it should behave as if it is not running (status=0).
+        - If the Spawner has not finished starting,
+          it should behave as if it is running (status=None).
+
+        Design assumptions about when `poll` may be called:
+
+        - On Hub launch: `poll` may be called before `start` when state is loaded on Hub launch.
+          `poll` should return exit status 0 (unknown) if the Spawner has not been initialized via
+          `load_state` or `start`.
+        - If `.start()` is async: `poll` may be called during any yielded portions of the `start`
+          process. `poll` should return None when `start` is yielded, indicating that the `start`
+          process has not yet completed.
+
         """
         raise NotImplementedError("Override in subclass. Must be a Tornado gen.coroutine.")
     


Test patch: