Repository: jupyterhub/jupyterhub

Issue:
Can we override `Authenticator.check_allowed` in jupyterhub/oauthenticator and assume authentication argument be passed?
In [oauthenticator #594](https://github.com/jupyterhub/oauthenticator/pull/594) we are currently planning to override the `check_allowed` as considered and proposed in https://github.com/jupyterhub/oauthenticator/pull/594#issuecomment-1519086377 and the comments below. The [`check_allowed` function signature](https://github.com/jupyterhub/jupyterhub/blob/4.0.0/jupyterhub/auth.py#L432-L438) is...

```python
    def check_allowed(self, username, authentication=None):
        """Check if a username is allowed to authenticate based on configuration

        Return True if username is allowed, False otherwise.
        No allowed_users set means any username is allowed.

        Names are normalized *before* being checked against the allowed set.
```

What came as a surprise to me now was that check_allowed was also being called by JupyterHub without passing the authentication dictionary (aka. `auth_model`), and having error messages related to specifically that `allowed_users` check failing.

It seems that also the [LocalAuthenticator](https://github.com/jupyterhub/jupyterhub/blob/main/jupyterhub/auth.py#L871-L875) class overrides `check_allowed`, and adds a check for beloning to either allowed_users or allowed_groups very similar to what we want to do. Our unique challenge is that we rely on being passed `authentication` aka `auth_model`, and that isn't passed at all times when check_allowed is used.

In practice it seems that only if the deprecated `JupyterHub.api_tokens` is used or if `JupyterHub.load_roles` is configured, and they involve user api tokens or similar, then `check_allowed` will be called without an authentication dictionary.

### Using `JupyterHub.load_roles`

[`JupyterHub.init_role_assignment`](https://github.com/jupyterhub/jupyterhub/blob/4.0.0/jupyterhub/app.py#L2274-L2295)

```python
    async def init_role_assignment(self):
        """
        This function loads config from `JupyterHub.load_roles`
        """
        # tokens are added separately
        kinds = ['users', 'services', 'groups']

        # ...

        for role_spec in self.load_roles:
        
            # ...
        
            # add users, services, and/or groups,
            # tokens need to be checked for permissions
            for kind in kinds:
                orm_role_bearers = []
                if kind in role_spec:
                    for name in role_spec[kind]:
                        if kind == 'users':
                            name = self.authenticator.normalize_username(name)
                            if not (
                                await maybe_future(
                                    self.authenticator.check_allowed(name, None)
                                )
                            ):
                                raise ValueError(
                                    f"Username {name} is not in Authenticator.allowed_users"
                                )
```

### Using `JupyterHub.api_tokens`

[`JupyterHub.api_tokens`](https://github.com/jupyterhub/jupyterhub/blob/4.0.0/jupyterhub/app.py#L1049-L1060)

```python
    api_tokens = Dict(
        Unicode(),
        help="""PENDING DEPRECATION: consider using services

        Dict of token:username to be loaded into the database.

        Allows ahead-of-time generation of API tokens for use by externally managed services,
        which authenticate as JupyterHub users.

        Consider using services for general services that talk to the JupyterHub API.
        """,
    ).tag(config=True)
```

[`JupyterHub.init_api_tokens`](https://github.com/jupyterhub/jupyterhub/blob/4.0.0/jupyterhub/app.py#L2345-L2349)

```python
    async def init_api_tokens(self):
        """Load predefined API tokens (for services) into database"""

        await self._add_tokens(self.service_tokens, kind='service')
        await self._add_tokens(self.api_tokens, kind='user')
        
        # ...
```

[`JupyterHub._add_token`](https://github.com/jupyterhub/jupyterhub/blob/4.0.0/jupyterhub/app.py#L2274-L2295)

```python
    async def _add_tokens(self, token_dict, kind):
        """Add tokens for users or services to the database"""
        if kind == 'user':
            Class = orm.User
        elif kind == 'service':
            Class = orm.Service
        else:
            raise ValueError("kind must be user or service, not %r" % kind)

        db = self.db
        for token, name in token_dict.items():
            if kind == 'user':
                name = self.authenticator.normalize_username(name)
                if not (
                    await maybe_future(self.authenticator.check_allowed(name, None))
                ):
                    raise ValueError(
                        "Token user name %r is not in Authenticator.allowed_users"
                        % name
                    )
                if not self.authenticator.validate_username(name):
                    raise ValueError("Token user name %r is not valid" % name)
```


Discussion:
You can't assume that it will always be passed. When users are added via configuration, the allow check is called to avoid creating users that aren't allowed elsewhere during the configuration process, which adds them to the allowed users set, etc. If the check was skipped, then users added via configuration would become allowed who shouldn't be.

This is specifically an issue for the per-user allow list. If using other allow checks _to the exclusion of the allowed_users set_, then these extra checks don't matter because adding users to the db doesn't affect whether they will be allowed later.

You _can_, however, assume that if `None` is given as the authentication info, that it's coming from startup configuration such as role assignment. So it would not be unreasonable to return True _if_ team auth is enabled and auth info is None. The failure mode of doing so would be:

1. user is on a team
2. team is granted access to jupyterhub
3. user is granted specific permission via role assignments
4. user is removed from a team, losing access to _authentication_, but the user keeps their role assignment

The allowed_user check is applied at _authentication time_ (i.e. login), not on every request, so this user who has had their permission revoked will no longer be able to login. However, any tokens they have been issued will still have access to the assigned roles. Arguably, this behavior matches the configuration - the user has been explicitly granted this permission and their login access has been revoked, but it doesn't make a huge amount of sense.

If historical background is helpful - for a long time, only the username was passed to the allow check. The authentication dict was added _when available_ to enable things like oauth team allow lists, but the checks were already present in more places than the auth could be made available.

I'm not quite sure what changes in JupyterHub would make this better. We could remove the allow check entirely in app startup. I'll need to think about the consequences there, but it should probably be the right behavior for e.g. GitHub team auth.

In general, there's an issue with token expiry and upstream user auth state changes, because a user's auth state (e.g. team membership) doesn't get updated until _that user_ logs in again. That's usually fine for privilege _escalation_, but privilege _revocation_ does not take effect promptly. It works fine for login and spawn, but not for token access to the API. If a deployment relies on things like teams auth, they should probably apply things like requiring shorter token expiry (and JupyterHub should make this easier), to limit the window where these can be out of sync.
> When users are added via configuration, the allow check is called to avoid creating users that aren't allowed elsewhere during the configuration process, _**which adds them to the allowed users set**_, etc.

Ah... I'm catching up with how things really works. Something that becomes clear now that I didn't understand was that:

1. Users in allowed_users not in db are added to the db
2. Users in db are added to the allowed_users, but only if allowed_users is truthy already via `Authenticator.add_users`

Okay so this means `allowed_users` when referenced by the authenticator in check_allowed etc, is either a set of all users configured by the user explicitly and the list of users in the database, or an empty set.

[`JupyterHub.init_users`](https://github.com/jupyterhub/jupyterhub/blob/4.0.0/jupyterhub/app.py#L1928-L2005)

```python
    async def init_users(self):
        # [...]

        # Notify authenticator of all users.
        # This ensures Authenticator.allowed_users is up-to-date with the database.
        # This lets .allowed_users be used to set up initial list,
        # but changes to the allowed_users set can occur in the database,
        # and persist across sessions.
        total_users = 0
        for user in db.query(orm.User):
            try:
                f = self.authenticator.add_user(user)
            except Exception:
                self.log.exception("Error adding user %s already in db", user.name)
                if self.authenticator.delete_invalid_users:
                    self.log.warning(
                        "Deleting invalid user %s from the Hub database", user.name
                    )
                    db.delete(user)
                else:
                    self.log.warning(
                        dedent(
                            """
                    You can set
                        c.Authenticator.delete_invalid_users = True
                    to automatically delete users from the Hub database that no longer pass
                    Authenticator validation,
                    such as when user accounts are deleted from the external system
                    without notifying JupyterHub.
                    """
                        )
                    )
```

[`Authenticator.add_user`](https://github.com/jupyterhub/jupyterhub/blob/4.0.0/jupyterhub/auth.py#L625-L648)

```python
    def add_user(self, user):
        """Hook called when a user is added to JupyterHub

        This is called:
         - When a user first authenticates
         - When the hub restarts, for all users.

        This method may be a coroutine.

        By default, this just adds the user to the allowed_users set.

        Subclasses may do more extensive things, such as adding actual unix users,
        but they should call super to ensure the allowed_users set is updated.

        Note that this should be idempotent, since it is called whenever the hub restarts
        for all users.

        Args:
            user (User): The User wrapper object
        """
        if not self.validate_username(user.name):
            raise ValueError("Invalid username: %s" % user.name)
        if self.allowed_users:
            self.allowed_users.add(user.name)
```

[`Authenticator.delete_user`](https://github.com/jupyterhub/jupyterhub/blob/4.0.0/jupyterhub/auth.py#L650-L659)

```python
    def delete_user(self, user):
        """Hook called when a user is deleted

        Removes the user from the allowed_users set.
        Subclasses should call super to ensure the allowed_users set is updated.

        Args:
            user (User): The User wrapper object
        """
        self.allowed_users.discard(user.name)
```
> If the check was skipped, then users added via configuration would become allowed who shouldn't be.

I need to understand this better, thinking about the specific situations. There was two situations when check_allowed was called:

- `init_api_tokens` -> `_add_tokens` (influenced by deprecated since v0.8 `JupyterHub.api_tokens` config)
- `init_role_assignment` (influenced by `JupyterHub.load_roles` config)

I'll disregard the first case with use of since very long deprecated `JupyterHub.api_tokens` for simplicity, focusing on the second case with `JupyterHub.load_roles` involving users. Here is an example [from jupyterhub docs](https://jupyterhub.readthedocs.io/en/stable/rbac/roles.html#defining-roles) on using this config.

```python
c.JupyterHub.load_roles = [
 {
   'description': 'Read-only user models',
   'name': 'reader',
   'scopes': ['read:users'],
   'services': ['external'],
   'users': ['maria', 'joe']
 }
]
```

So if we make `check_allowed` respond True when passed no authentication data, then we influence this code, which means the user will be added to the database, which means that on subsequent startups, it will also be added to the added to truthy allowed_users sets...

https://github.com/jupyterhub/jupyterhub/blob/0e4deec714a30729d11e7d0b1ce359f65faaf6af/jupyterhub/app.py#L2193-L2219

### Situations to drive discussion - GitHubOAuthenticator.allowed_organization

The current state of things when for example using GitHubOAuthenticator configured with `allowed_organization` is that `check_allowed` will ignore and the authentication logic will consider it. If one would configure allowed_users together with allowed_organization, then one would need to configure allowed_users to a set of users inside the allowed_organization anyway, making it pointless to have allowed_organization.

### Understanding of issues summarized

- `check_allowed` is called on jupyterhub startup without authentication info passed when referencing users via `JupyterHub.load_roles`, `JupyterHub.load_groups`, or deprecated `JupyterHub.api_tokens`.
- Getting authentication info _could require_ approval from the authenticated user, and just can't be something we assume to have when starting up jupyterhub that parses `load_roles` and wants to call `check_allowed`.
- This makes use of `check_allowed` without an authentication dict problematic, because we can only get a signal if the user is allowed, not know for sure no matter what.

### Key question in my mind

I want to articulate why check_allowed should be called at all. Is it because otherwise load_roles and load_groups would influence a configured allowed_users list because users are added to the database, and after restart, they get unexpectedly allowed because of it?

@minrk could you try help refine the need for check_allowed to be called when using `load_roles` and `load_groups`?
The general idea is that "when attempting to add a user, check if they are allowed", so we perform the check when users are added. load_roles/groups are places where users are added. It's a way to ensure that configuration is self-consistent. For example, without check_allowed (or with a short-circuit that returns True without auth info), you could grant access to JupyterHUb to a user who cannot login, because check_allowed at login time would fail. If they _never_ have access, this is an inconsistency of little consequence (a user exists who can never login). But if they have had access in the past, any api tokens they have will still be usable, but they will never be able to login again.

_but_! since those who specify load_roles, etc. config necessarily have control of allowed_users config, we can also implicitly _allow_  any user added in this way.
> For example, without check_allowed (or with a short-circuit that returns True without auth info), you could grant access to JupyterHUb to a user who cannot login, because check_allowed at login time would fail. If they never have access, this is an inconsistency of little consequence (a user exists who can never login).

I agree, this is acceptable.

> But if they have had access in the past, any api tokens they have will still be usable, but they will never be able to login again.

Okay hmm...

> _but_! since those who specify load_roles, etc. config necessarily have control of allowed_users config, we can also implicitly _allow_ any user added in this way.

Hmmm... Are you saying that it could make sense to call `authenticator.add_user()` if they are found in load_roles or load_groups?
Yes, I'm saying that if we treat the presence of any user in config as an implicit addition to `allowed_users`. Rather than _enforcing_ that config is consistent, it's _ensuring_ that it's consistent (or assuming it). Removing the check accomplishes this.

i.e. we go from:

>  if you add a user anywhere in your config that's not consistent with allowed_users, we raise to tell you that your config doesn't make sense.

to

> if you add a user anywhere in your config, that user should be allowed access to jupyterhub.

It will still be possible to get into an inconsistent state with 'access' vs 'login', depending on check_allowed when using external team membership, etc., but that's not really different from today - it's just not evaluated at hub startup time anymore.


What will be lost is simple cases of usually catching typos in usernames, but that's perhaps not the most important thing.
### Can we get state changes by restarting?

I end up thinking about a case where a user is created and added to the database. Should it always go hand in hand with calling `authenticator.add_user`? Right now, load_roles and load_groups creates users if check_allowed returns true, but doesn't call `authenticator.add_user`. But, it will be called after restart because they are part of the database, and they will be added to `allowed_users` if it was set at all at that point as well.

Should we make a call to `authenticator.add_user` consistently when creating a user to make the state consistent with the state it arrives at after a restart?
> What will be lost is simple cases of usually catching typos in usernames, but that's perhaps not the most important thing.

We could still call normalize and validate username on startup right? These function doesn't require authentication dictionaries.
> I end up thinking about a case where a user is created and added to the database. Should it always go hand in hand with calling authenticator.add_user? 

Yes, I believe so.


> Should we make a call to authenticator.add_user consistently when creating a user to make the state consistent with the state it arrives at after a restart?

Yes, and definitely do need to if we change the behavior. As it is now, only allowed users get past this step, so `add_user` should have already been called for that user. But if we change it so that users can be created at this point, we should call `add_user`, yes. There may be cases for more complex implementations of allowing users where add_user is missed during creation via config for previously undeclared users.

> We could still call normalize and validate username on startup right?

Yes, and we should. That's not relevant to the issue, though - e.g. a mismatch in spelling between `allowed_users` and a role assignment.

Erik:
> if you add a user anywhere in your config, that user should be allowed access to jupyterhub.

Min:
> It will still be possible to get into an inconsistent state with 'access' vs 'login', depending on check_allowed when using external team membership, etc., but that's not really different from today - it's just not evaluated at hub startup time anymore.
>
> What will be lost is simple cases of usually catching typos in usernames, but that's perhaps not the most important thing.

Erik:
> We could still call normalize and validate username on startup right? These function doesn't require authentication dictionaries.

Min:
> Yes, and we should. That's not relevant to the issue, though - e.g. a mismatch in spelling between allowed_users and a role assignment.

---

Okay I figure we could log a info/warning about users being added to the database because it was listed in load_roles / load_groups, but it would only present once I guess, as after that the user would be in the database.

There is a info log about this for users added via load_roles, but not for users added via load_groups.
Wieee thank you @minrk for discussing this with me!

Okay, here are things I've distilled from the conversation so far!

- Assumption: In https://github.com/jupyterhub/oauthenticator/pull/594, me and @GeorgianaElena proceed with plans on overriding `check_allowed`, and we make it return True when the authentication dictionary isn't passed (`None`). By doing this, we make it not crash when `load_roles` or `load_groups` is used to create users.
- Question: do we stop calling `check_allowed` in jupyterhub itself from app.py without an authentication dictionary?
  - Assumption: if we do, we ensure authenticator.add_user is called when a new user is created
  - Question: is this a breaking change, and how do we describe it?
- Assumption: we should print a log info about creating a new user via load_groups like done for load_roles

Resolving patch:
diff --git a/jupyterhub/app.py b/jupyterhub/app.py
--- a/jupyterhub/app.py
+++ b/jupyterhub/app.py
@@ -2041,21 +2041,20 @@ async def init_users(self):
 
         TOTAL_USERS.set(total_users)
 
-    async def _get_or_create_user(self, username):
+    async def _get_or_create_user(self, username, hint):
         """Create user if username is found in config but user does not exist"""
-        if not (await maybe_future(self.authenticator.check_allowed(username, None))):
-            raise ValueError(
-                "Username %r is not in Authenticator.allowed_users" % username
-            )
         user = orm.User.find(self.db, name=username)
         if user is None:
             if not self.authenticator.validate_username(username):
                 raise ValueError("Username %r is not valid" % username)
-            self.log.info(f"Creating user {username}")
+            self.log.info(f"Creating user {username} found in {hint}")
             user = orm.User(name=username)
             self.db.add(user)
             roles.assign_default_roles(self.db, entity=user)
             self.db.commit()
+            f = self.authenticator.add_user(user)
+            if f:
+                await maybe_future(f)
         return user
 
     async def init_groups(self):
@@ -2064,7 +2063,9 @@ async def init_groups(self):
 
         if self.authenticator.manage_groups and self.load_groups:
             raise ValueError("Group management has been offloaded to the authenticator")
+
         for name, contents in self.load_groups.items():
+            self.log.debug("Loading group %s", name)
             group = orm.Group.find(db, name)
 
             if group is None:
@@ -2080,9 +2081,11 @@ async def init_groups(self):
             if 'users' in contents:
                 for username in contents['users']:
                     username = self.authenticator.normalize_username(username)
-                    user = await self._get_or_create_user(username)
+                    user = await self._get_or_create_user(
+                        username, hint=f"group: {name}"
+                    )
                     if group not in user.groups:
-                        self.log.debug(f"Adding user {username} to  group {name}")
+                        self.log.debug(f"Adding user {username} to group {name}")
                         group.users.append(user)
 
             if 'properties' in contents:
@@ -2110,8 +2113,9 @@ async def init_role_creation(self):
         roles_with_new_permissions = []
         for role_spec in self.load_roles:
             role_name = role_spec['name']
+            self.log.debug("Loading role %s", role_name)
             if role_name in default_roles_dict:
-                self.log.debug(f"Overriding default role {role_name}")
+                self.log.debug("Overriding default role %s", role_name)
                 # merge custom role spec with default role spec when overriding
                 # so the new role can be partially defined
                 default_role_spec = default_roles_dict.pop(role_name)
@@ -2198,34 +2202,33 @@ async def init_role_assignment(self):
                     for name in role_spec[kind]:
                         if kind == 'users':
                             name = self.authenticator.normalize_username(name)
-                            if not (
-                                await maybe_future(
-                                    self.authenticator.check_allowed(name, None)
-                                )
-                            ):
-                                raise ValueError(
-                                    f"Username {name} is not in Authenticator.allowed_users"
-                                )
                         Class = orm.get_class(kind)
                         orm_obj = Class.find(db, name)
                         if orm_obj is not None:
                             orm_role_bearers.append(orm_obj)
                         else:
-                            self.log.info(
-                                f"Found unexisting {kind} {name} in role definition {role_name}"
-                            )
                             if kind == 'users':
-                                orm_obj = await self._get_or_create_user(name)
+                                orm_obj = await self._get_or_create_user(
+                                    name, hint=f"role: {role_name}"
+                                )
                                 orm_role_bearers.append(orm_obj)
                             elif kind == 'groups':
+                                self.log.info(
+                                    f"Creating group {name} found in role: {role_name}"
+                                )
                                 group = orm.Group(name=name)
                                 db.add(group)
                                 db.commit()
                                 orm_role_bearers.append(group)
-                            else:
+                            elif kind == "services":
                                 raise ValueError(
-                                    f"{kind} {name} defined in config role definition {role_name} but not present in database"
+                                    f"Found undefined service {name} in role {role_name}. Define it first in c.JupyterHub.services."
                                 )
+                            else:
+                                # this can't happen now, but keep the `else` in case we introduce a problem
+                                # in the declaration of `kinds` above
+                                raise ValueError(f"Unhandled role member kind: {kind}")
+
                         # Ensure all with admin role have admin flag
                         if role_name == 'admin':
                             orm_obj.admin = True
@@ -2285,20 +2288,12 @@ async def _add_tokens(self, token_dict, kind):
         for token, name in token_dict.items():
             if kind == 'user':
                 name = self.authenticator.normalize_username(name)
-                if not (
-                    await maybe_future(self.authenticator.check_allowed(name, None))
-                ):
-                    raise ValueError(
-                        "Token user name %r is not in Authenticator.allowed_users"
-                        % name
-                    )
                 if not self.authenticator.validate_username(name):
                     raise ValueError("Token user name %r is not valid" % name)
             if kind == 'service':
                 if not any(service["name"] == name for service in self.services):
                     self.log.warning(
-                        "Warning: service '%s' not in services, creating implicitly. It is recommended to register services using services list."
-                        % name
+                        f"service {name} not in services, creating implicitly. It is recommended to register services using services list."
                     )
             orm_token = orm.APIToken.find(db, token)
             if orm_token is None:


Test patch:
diff --git a/jupyterhub/tests/test_app.py b/jupyterhub/tests/test_app.py
--- a/jupyterhub/tests/test_app.py
+++ b/jupyterhub/tests/test_app.py
@@ -251,7 +251,6 @@ async def test_load_groups(tmpdir, request):
     hub.init_db()
     db = hub.db
     await hub.init_role_creation()
-
     await hub.init_users()
     await hub.init_groups()
 
@@ -435,3 +434,42 @@ async def start(self):
         assert hub.argv == sys_argv[1:]
     else:
         assert hub.argv == argv
+
+
+async def test_user_creation(tmpdir, request):
+    allowed_users = {"in-allowed", "in-group-in-allowed", "in-role-in-allowed"}
+    groups = {
+        "group": {
+            "users": ["in-group", "in-group-in-allowed"],
+        }
+    }
+    roles = [
+        {
+            "name": "therole",
+            "users": ["in-role", "in-role-in-allowed"],
+        }
+    ]
+
+    cfg = Config()
+    cfg.Authenticator.allowed_users = allowed_users
+    cfg.JupyterHub.load_groups = groups
+    cfg.JupyterHub.load_roles = roles
+    ssl_enabled = getattr(request.module, "ssl_enabled", False)
+    kwargs = dict(config=cfg)
+    if ssl_enabled:
+        kwargs['internal_certs_location'] = str(tmpdir)
+    hub = MockHub(**kwargs)
+    hub.init_db()
+
+    await hub.init_role_creation()
+    await hub.init_role_assignment()
+    await hub.init_users()
+    await hub.init_groups()
+    assert hub.authenticator.allowed_users == {
+        "admin",  # added by default config
+        "in-allowed",
+        "in-group-in-allowed",
+        "in-role-in-allowed",
+        "in-group",
+        "in-role",
+    }