Repository: jupyterhub/jupyterhub

Issue:
Deleting OS users causes Jupyterhub server to fail to start
Jupyterhub does not properly check if a user exists when it attempts to start. If an OS user is deleted, Jupyterhub will crash until those users have been removed from Jupyterhub's database. 

**How to reproduce the issue**

* Install Jupyterhub according to https://github.com/jupyterhub/jupyterhub/wiki/Using-sudo-to-run-JupyterHub-without-root-privileges. 
* Create a test user (`useradd test`). 
* Log in, make a notebook or three with said test user. 
* Delete test user (`userdel -r test`).
* Restart jupyterhub.

**What you expected to happen**

Jupyterhub should start successfully.

**What actually happens**

Jupyterhub will now crash on start.

**Share what version of JupyterHub you are using**

0.7.2

**HOW TO FIX**

Check for non-existent users on Jupyterhub initialization, delete them from Jupyterhub's `users` table.

```
sqlite3 jupyterhub.sqlite
DELETE FROM users WHERE name LIKE 'test%';
.quit
# jupyterhub now starts normally
```

**Logs (from a jupyterhub systemd service)**

Apr 06 14:00:07 some_host.ca sudo[8433]: jupyterhub : unknown user: testme
Apr 06 14:00:07 some_host.ca jupyterhub[8429]: sudo: unknown user: testme
Apr 06 14:00:07 some_host.ca jupyterhub[8429]: sudo: unable to initialize policy plugin
Apr 06 14:00:07 some_host.ca jupyterhub[8429]: [E 2017-04-06 14:00:07.756 JupyterHub app:1527]
Apr 06 14:00:07 some_host.ca jupyterhub[8429]: Traceback (most recent call last):
Apr 06 14:00:07 some_host.ca jupyterhub[8429]: File "/opt/python/anaconda3/lib/python3.6/site-packages/jupyterhub/app.py", line 1524, in launch_instance_async
Apr 06 14:00:07 some_host.ca jupyterhub[8429]: yield self.initialize(argv)
Apr 06 14:00:07 some_host.ca jupyterhub[8429]: File "/opt/python/anaconda3/lib/python3.6/types.py", line 184, in throw
Apr 06 14:00:07 some_host.ca jupyterhub[8429]: return self.__wrapped.throw(tp, *rest)
Apr 06 14:00:07 some_host.ca jupyterhub[8429]: File "/opt/python/anaconda3/lib/python3.6/site-packages/jupyterhub/app.py", line 1315, in initialize
Apr 06 14:00:07 some_host.ca jupyterhub[8429]: yield self.init_spawners()
Apr 06 14:00:07 some_host.ca jupyterhub[8429]: File "/opt/python/anaconda3/lib/python3.6/site-packages/jupyterhub/app.py", line 1087, in init_spawners
Apr 06 14:00:07 some_host.ca jupyterhub[8429]: status = yield spawner.poll()
Apr 06 14:00:07 some_host.ca jupyterhub[8429]: File "/opt/python/anaconda3/lib/python3.6/site-packages/jupyterhub/spawner.py", line 834, in poll
Apr 06 14:00:07 some_host.ca jupyterhub[8429]: alive = yield self._signal(0)
Apr 06 14:00:07 some_host.ca jupyterhub[8429]: File "/opt/python/anaconda3/lib/python3.6/site-packages/sudospawner/spawner.py", line 90, in _signal
Apr 06 14:00:07 some_host.ca jupyterhub[8429]: reply = yield self.do('kill', pid=self.pid, signal=sig)
Apr 06 14:00:07 some_host.ca jupyterhub[8429]: File "/opt/python/anaconda3/lib/python3.6/site-packages/sudospawner/spawner.py", line 69, in do
Apr 06 14:00:07 some_host.ca jupyterhub[8429]: data_str = data_str[data_str.index('{'):data_str.rindex('}')+1]
Apr 06 14:00:07 some_host.ca jupyterhub[8429]: ValueError: substring not found
Apr 06 14:00:07 some_host.ca systemd[1]: jupyterhub.service: main process exited, code=exited, status=1/FAILURE



Discussion:
Hi @jstaf,

Thanks for the well detailed issue report. Sorry you ran into this.

Currently, JupyterHub does not check for this use case which is described in the [documentation](https://jupyterhub.readthedocs.io/en/latest/getting-started.html#add-or-remove-users-from-the-hub) and summarized below:

> After starting the Hub once, it is not sufficient to remove a user from the whitelist in your config file. You must also remove the user from the Hub’s database, either by deleting the user from the admin page, or you can clear the jupyterhub.sqlite database and start fresh.

I'm going to go ahead and label this as an "enhancement" for better "user experience".
The severity of the failure is specific to sudospawner, which relies on becoming the user in order to check if the user is running. Other Spawners, such as the default one, ought to have no such problems.

If SudoSpawner checked for user existence before`.poll()`, it would also have no such issue.

During JupyterHub startup, the following happens:

- iterate through users in db, check their validity
  - LocalAuthenticators check if corresponding system users exist
  - if it doesn't:
    - if create_system_users: create system user
    - else: *warn* that system user doesn't exist ⬅️
  - instantiate a Spawner and call `.poll()` to check if each user's server is still running. ⬅️ this is where SudoSpawner fails

At the first arrow, users could be deleted if they don't exist, instead of a simple warning. I'm hesitant to do that by default, because a bug in user configuration could trigger deleting a bunch of users from the database. But there should be an easy way, perhaps opt-in like `create_system_users`, to cleanup the users that should no longer exist.
Hmmmmm... although you do have documentation buried rather deeply saying what needs to happen (also, that's a broken link), Jupyterhub needs to be able to gracefully handle normal sysadmin activities like deleting users, regardless of which spawner gets used. 

I think opt-in user removal is a good suggestion (and the default behavior of sudospawner, where failure to delete users results in a crash).
@jstaf Sorry about the copy-pasta error in the link. I've fixed it above. 

>I think opt-in user removal is a good suggestion (and the default behavior of sudospawner, where failure to delete users results in a crash).

Yeah, I think this suggestion (`create_system_users`) from @minrk makes good sense. Thanks for sharing your experience with us too.

Resolving patch:
diff --git a/jupyterhub/app.py b/jupyterhub/app.py
--- a/jupyterhub/app.py
+++ b/jupyterhub/app.py
@@ -954,11 +954,20 @@ def init_users(self):
             try:
                 yield gen.maybe_future(self.authenticator.add_user(user))
             except Exception:
-                # TODO: Review approach to synchronize whitelist with db
-                # known cause of the exception is a user who has already been removed from the system
-                # but the user still exists in the hub's user db
                 self.log.exception("Error adding user %r already in db", user.name)
-        db.commit()  # can add_user touch the db?
+                if self.authenticator.delete_invalid_users:
+                    self.log.warning("Deleting invalid user %r from the Hub database", user.name)
+                    db.delete(user)
+                else:
+                    self.log.warning(dedent("""
+                    You can set
+                        c.Authenticator.delete_invalid_users = True
+                    to automatically delete users from the Hub database that no longer pass
+                    Authenticator validation,
+                    such as when user accounts are deleted from the external system
+                    without notifying JupyterHub.
+                    """))
+        db.commit()
 
         # The whitelist set and the users in the db are now the same.
         # From this point on, any user changes should be done simultaneously
@@ -1151,7 +1160,14 @@ def user_stopped(user):
             self.users[orm_user.id] = user = User(orm_user, self.tornado_settings)
             self.log.debug("Loading state for %s from db", user.name)
             spawner = user.spawner
-            status = yield spawner.poll()
+            status = 0
+            if user.server:
+                try:
+                    status = yield spawner.poll()
+                except Exception:
+                    self.log.exception("Failed to poll spawner for %s, assuming the spawner is not running.", user.name)
+                    status = -1
+
             if status is None:
                 self.log.info("%s still running", user.name)
                 spawner.add_poll_callback(user_stopped, user)
diff --git a/jupyterhub/auth.py b/jupyterhub/auth.py
--- a/jupyterhub/auth.py
+++ b/jupyterhub/auth.py
@@ -125,6 +125,23 @@ def validate_username(self, username):
         """
     ).tag(config=True)
 
+    delete_invalid_users = Bool(False,
+        help="""Delete any users from the database that do not pass validation
+
+        When JupyterHub starts, `.add_user` will be called
+        on each user in the database to verify that all users are still valid.
+
+        If `delete_invalid_users` is True,
+        any users that do not pass validation will be deleted from the database.
+        Use this if users might be deleted from an external system,
+        such as local user accounts.
+
+        If False (default), invalid users remain in the Hub's database
+        and a warning will be issued.
+        This is the default to avoid data loss due to config changes.
+        """
+    )
+
     def normalize_username(self, username):
         """Normalize the given username and return it
 


Test patch: