Repository: jupyterhub/jupyterhub

Issue:
Problems overriding the default roles
<!-- Thank you for contributing. These HTML comments will not render in the issue, but you can delete them once you've read them if you prefer! -->

### Bug description
It seems that overriding the role `user` is not functioning correctly. Specifically, on jh 2.0.0 if I set up the following role config (see [here](https://discourse.jupyter.org/t/plans-on-bringing-rtc-to-jupyterhub/9813/11?u=anton_akhmerov) for motivation)

```python
c.JupyterHub.load_roles = [
    {
        "name": "user",
        "description": "Allow users to access the shared server in addition to default perms",
        "scopes": ["self", "access:servers!user=shared"],
    }
]
```

Now after starting the hub, any non-admin user that existed on the hub before the start is unable to launch their server and sees a 500 error:

![image](https://user-images.githubusercontent.com/2069677/144727721-6f6b5578-41f6-4704-9ffa-11c08428f7fc.png)

After deleting and recreating the user from the hub admin panel, everything seems to function normally.


Discussion:
@akhmerov thanks for the report! I'll dig into this.

In the meantime, do you have the log line that starts with `Token requesting role`? It should show specifically what permissions are conflicting.
Sure:
```
[W 2021-12-06 13:19:17.896 JupyterHub roles:436] Token requesting role server with scopes not held by owner shared: {'access:servers', 'users:activity', 'read:users:activity'}
```
Have you specified the `server` role as well?
No, what wrote above is the complete role spec. Is it necessary to set something for the server? I thought `server` inherits the subset of scopes. Plus if it was necessary to specify the server then it wouldn't explain why deleting and readding the user changes the behavior.
>  Is it necessary to set something for the server? 

You shouldn't need to, I'm just trying to track down the source of the problem, since I haven't been able to reproduce it myself just yet.

The relevant details:

1. `user` role defines the permissions held by all JupyterHub users
2. `server` role defines the permissions of the $JUPYTERHUB_API_TOKEN that the server itself uses. The [default][] is
    ```python
    [
                'users:activity!user',
                'access:servers!user',
    ],
    ```
   where `!user` is an alias for `!user=$name` for the _owner_ of the server

3. when issuing a token owned by a user, JupyterHub [checks][] that the token isn't being given permissions that its owner doesn't have.

What you are experiencing is the token issued during server startup for hte user `shared` is requesting the role `server`, but the user `shared` doesn't have a superset of the the requested permissions for starting a server.

Here's a question: is the `shared` user in your `allowed_users` list? I believe that would cause the issue, because then the shared user might not have any role assigned. How are you starting the shared server?

[default]: https://github.com/jupyterhub/jupyterhub/blob/2.0.0/jupyterhub/roles.py#L50-L54
[checks]: https://github.com/jupyterhub/jupyterhub/blob/2.0.0/jupyterhub/roles.py#L402-L439
The problem doesn't only apply to the `shared` user. Rather all users except for admins encounter this problem with their own servers. It appears both if I attempt to launch a user's server via the admin panel or if a user tries to do it on their own.

Hmm, we don't have `allowed_users` explicitly specified in the config. Most users are created by the oauthenticator, and `shared` was created via the admin interface.
Ah, gotcha. and the issue mainly occurs on Hub restart? Then I think I know where to look.

My guess is that users are being stripped of their membership in the `user` role because you have defined a custom user role, but not specified any members. This would be the right thing to do if you had specified an explicit empty list (`"users": []`), but perhaps it behaves the same if the membership is simply omitted.

There are too many places user creation can happen!
Indeed, the problem occurs after hub restart with every non-admin user that is in the database before the hub has restarted. I have no custom roles defined, only `user` and no non-admin users defined in the config.

Resolving patch:
diff --git a/jupyterhub/app.py b/jupyterhub/app.py
--- a/jupyterhub/app.py
+++ b/jupyterhub/app.py
@@ -2015,14 +2015,25 @@ async def init_groups(self):
 
     async def init_role_creation(self):
         """Load default and predefined roles into the database"""
-        self.log.debug('Loading default roles to database')
+        self.log.debug('Loading roles into database')
         default_roles = roles.get_default_roles()
         config_role_names = [r['name'] for r in self.load_roles]
 
-        init_roles = default_roles
+        default_roles_dict = {role["name"]: role for role in default_roles}
+        init_roles = []
         roles_with_new_permissions = []
         for role_spec in self.load_roles:
             role_name = role_spec['name']
+            if role_name in default_roles_dict:
+                self.log.debug(f"Overriding default role {role_name}")
+                # merge custom role spec with default role spec when overriding
+                # so the new role can be partially defined
+                default_role_spec = default_roles_dict.pop(role_name)
+                merged_role_spec = {}
+                merged_role_spec.update(default_role_spec)
+                merged_role_spec.update(role_spec)
+                role_spec = merged_role_spec
+
             # Check for duplicates
             if config_role_names.count(role_name) > 1:
                 raise ValueError(
@@ -2033,10 +2044,13 @@ async def init_role_creation(self):
             old_role = orm.Role.find(self.db, name=role_name)
             if old_role:
                 if not set(role_spec['scopes']).issubset(old_role.scopes):
-                    app_log.warning(
+                    self.log.warning(
                         "Role %s has obtained extra permissions" % role_name
                     )
                     roles_with_new_permissions.append(role_name)
+
+        # make sure we load any default roles not overridden
+        init_roles = list(default_roles_dict.values()) + init_roles
         if roles_with_new_permissions:
             unauthorized_oauth_tokens = (
                 self.db.query(orm.APIToken)
@@ -2048,7 +2062,7 @@ async def init_role_creation(self):
                 .filter(orm.APIToken.client_id != 'jupyterhub')
             )
             for token in unauthorized_oauth_tokens:
-                app_log.warning(
+                self.log.warning(
                     "Deleting OAuth token %s; one of its roles obtained new permissions that were not authorized by user"
                     % token
                 )
@@ -2056,14 +2070,19 @@ async def init_role_creation(self):
             self.db.commit()
 
         init_role_names = [r['name'] for r in init_roles]
-        if not orm.Role.find(self.db, name='admin'):
+        if (
+            self.db.query(orm.Role).first() is None
+            and self.db.query(orm.User).first() is not None
+        ):
+            # apply rbac-upgrade default role assignment if there are users in the db,
+            # but not any roles
             self._rbac_upgrade = True
         else:
             self._rbac_upgrade = False
         for role in self.db.query(orm.Role).filter(
             orm.Role.name.notin_(init_role_names)
         ):
-            app_log.info(f"Deleting role {role.name}")
+            self.log.warning(f"Deleting role {role.name}")
             self.db.delete(role)
         self.db.commit()
         for role in init_roles:
@@ -2079,71 +2098,89 @@ async def init_role_assignment(self):
         if config_admin_users:
             for role_spec in self.load_roles:
                 if role_spec['name'] == 'admin':
-                    app_log.warning(
+                    self.log.warning(
                         "Configuration specifies both admin_users and users in the admin role specification. "
                         "If admin role is present in config, c.Authenticator.admin_users should not be used."
                     )
-                    app_log.info(
+                    self.log.info(
                         "Merging admin_users set with users list in admin role"
                     )
                     role_spec['users'] = set(role_spec.get('users', []))
                     role_spec['users'] |= config_admin_users
-        self.log.debug('Loading predefined roles from config file to database')
+        self.log.debug('Loading role assignments from config')
         has_admin_role_spec = {role_bearer: False for role_bearer in admin_role_objects}
-        for predef_role in self.load_roles:
-            predef_role_obj = orm.Role.find(db, name=predef_role['name'])
-            if predef_role['name'] == 'admin':
+        for role_spec in self.load_roles:
+            role = orm.Role.find(db, name=role_spec['name'])
+            role_name = role_spec["name"]
+            if role_name == 'admin':
                 for kind in admin_role_objects:
-                    has_admin_role_spec[kind] = kind in predef_role
+                    has_admin_role_spec[kind] = kind in role_spec
                     if has_admin_role_spec[kind]:
-                        app_log.info(f"Admin role specifies static {kind} list")
+                        self.log.info(f"Admin role specifies static {kind} list")
                     else:
-                        app_log.info(
+                        self.log.info(
                             f"Admin role does not specify {kind}, preserving admin membership in database"
                         )
             # add users, services, and/or groups,
             # tokens need to be checked for permissions
             for kind in kinds:
                 orm_role_bearers = []
-                if kind in predef_role.keys():
-                    for bname in predef_role[kind]:
+                if kind in role_spec:
+                    for name in role_spec[kind]:
                         if kind == 'users':
-                            bname = self.authenticator.normalize_username(bname)
+                            name = self.authenticator.normalize_username(name)
                             if not (
                                 await maybe_future(
-                                    self.authenticator.check_allowed(bname, None)
+                                    self.authenticator.check_allowed(name, None)
                                 )
                             ):
                                 raise ValueError(
-                                    "Username %r is not in Authenticator.allowed_users"
-                                    % bname
+                                    f"Username {name} is not in Authenticator.allowed_users"
                                 )
                         Class = orm.get_class(kind)
-                        orm_obj = Class.find(db, bname)
+                        orm_obj = Class.find(db, name)
                         if orm_obj is not None:
                             orm_role_bearers.append(orm_obj)
                         else:
-                            app_log.info(
-                                f"Found unexisting {kind} {bname} in role definition {predef_role['name']}"
+                            self.log.info(
+                                f"Found unexisting {kind} {name} in role definition {role_name}"
                             )
                             if kind == 'users':
-                                orm_obj = await self._get_or_create_user(bname)
+                                orm_obj = await self._get_or_create_user(name)
                                 orm_role_bearers.append(orm_obj)
                             elif kind == 'groups':
-                                group = orm.Group(name=bname)
+                                group = orm.Group(name=name)
                                 db.add(group)
                                 db.commit()
                                 orm_role_bearers.append(group)
                             else:
                                 raise ValueError(
-                                    f"{kind} {bname} defined in config role definition {predef_role['name']} but not present in database"
+                                    f"{kind} {name} defined in config role definition {role_name} but not present in database"
                                 )
                         # Ensure all with admin role have admin flag
-                        if predef_role['name'] == 'admin':
+                        if role_name == 'admin':
                             orm_obj.admin = True
-                setattr(predef_role_obj, kind, orm_role_bearers)
+                    # explicitly defined list
+                    # ensure membership list is exact match (adds and revokes permissions)
+                    setattr(role, kind, orm_role_bearers)
+                else:
+                    # no defined members
+                    # leaving 'users' undefined in overrides of the default 'user' role
+                    # should not clear membership on startup
+                    # since allowed users could be managed by the authenticator
+                    if kind == "users" and role_name == "user":
+                        # Default user lists can be managed by the Authenticator,
+                        # if unspecified in role config
+                        pass
+                    else:
+                        # otherwise, omitting a member category is equivalent to specifying an empty list
+                        setattr(role, kind, [])
+
         db.commit()
         if self.authenticator.allowed_users:
+            self.log.debug(
+                f"Assigning {len(self.authenticator.allowed_users)} allowed_users to the user role"
+            )
             allowed_users = db.query(orm.User).filter(
                 orm.User.name.in_(self.authenticator.allowed_users)
             )
@@ -2160,8 +2197,8 @@ async def init_role_assignment(self):
         db.commit()
         # make sure that on hub upgrade, all users, services and tokens have at least one role (update with default)
         if getattr(self, '_rbac_upgrade', False):
-            app_log.warning(
-                "No admin role found; assuming hub upgrade. Initializing default roles for all entities"
+            self.log.warning(
+                "No roles found; assuming hub upgrade. Initializing default roles for all entities"
             )
             for kind in kinds:
                 roles.check_for_default_roles(db, kind)
diff --git a/jupyterhub/roles.py b/jupyterhub/roles.py
--- a/jupyterhub/roles.py
+++ b/jupyterhub/roles.py
@@ -239,26 +239,6 @@ def _check_scopes(*args, rolename=None):
                 )
 
 
-def _overwrite_role(role, role_dict):
-    """Overwrites role's description and/or scopes with role_dict if role not 'admin'"""
-    for attr in role_dict.keys():
-        if attr == 'description' or attr == 'scopes':
-            if role.name == 'admin':
-                admin_role_spec = [
-                    r for r in get_default_roles() if r['name'] == 'admin'
-                ][0]
-                if role_dict[attr] != admin_role_spec[attr]:
-                    raise ValueError(
-                        'admin role description or scopes cannot be overwritten'
-                    )
-            else:
-                if role_dict[attr] != getattr(role, attr):
-                    setattr(role, attr, role_dict[attr])
-                    app_log.info(
-                        'Role %r %r attribute has been changed', role.name, attr
-                    )
-
-
 _role_name_pattern = re.compile(r'^[a-z][a-z0-9\-_~\.]{1,253}[a-z0-9]$')
 
 
@@ -293,6 +273,17 @@ def create_role(db, role_dict):
     description = role_dict.get('description')
     scopes = role_dict.get('scopes')
 
+    if name == "admin":
+        for _role in get_default_roles():
+            if _role["name"] == "admin":
+                admin_spec = _role
+                break
+        for key in ["description", "scopes"]:
+            if key in role_dict and role_dict[key] != admin_spec[key]:
+                raise ValueError(
+                    f"Cannot override admin role admin.{key} = {role_dict[key]}"
+                )
+
     # check if the provided scopes exist
     if scopes:
         _check_scopes(*scopes, rolename=role_dict['name'])
@@ -306,8 +297,22 @@ def create_role(db, role_dict):
         if role_dict not in default_roles:
             app_log.info('Role %s added to database', name)
     else:
-        _overwrite_role(role, role_dict)
-
+        for attr in ["description", "scopes"]:
+            try:
+                new_value = role_dict[attr]
+            except KeyError:
+                continue
+            old_value = getattr(role, attr)
+            if new_value != old_value:
+                setattr(role, attr, new_value)
+                app_log.info(
+                    f'Role attribute {role.name}.{attr} has been changed',
+                )
+                app_log.debug(
+                    f'Role attribute {role.name}.{attr} changed from %r to %r',
+                    old_value,
+                    new_value,
+                )
     db.commit()
 
 


Test patch:
diff --git a/jupyterhub/tests/test_roles.py b/jupyterhub/tests/test_roles.py
--- a/jupyterhub/tests/test_roles.py
+++ b/jupyterhub/tests/test_roles.py
@@ -1183,24 +1183,36 @@ async def test_no_admin_role_change():
         await hub.init_role_creation()
 
 
-async def test_user_config_respects_memberships():
-    role_spec = [
-        {
-            'name': 'user',
-            'scopes': ['self', 'shutdown'],
-        }
-    ]
-    user_names = ['eddy', 'carol']
-    hub = MockHub(load_roles=role_spec)
+@pytest.mark.parametrize(
+    "in_db, role_users, allowed_users, expected_members",
+    [
+        # users in the db, not specified in custom user role
+        # no change to membership
+        (["alpha", "beta"], None, None, ["alpha", "beta"]),
+        # allowed_users is additive, not strict
+        (["alpha", "beta"], None, {"gamma"}, ["alpha", "beta", "gamma"]),
+        # explicit empty revokes all assignments
+        (["alpha", "beta"], [], None, []),
+        # explicit value is respected exactly
+        (["alpha", "beta"], ["alpha", "gamma"], None, ["alpha", "gamma"]),
+    ],
+)
+async def test_user_role_from_config(
+    in_db, role_users, allowed_users, expected_members
+):
+    role_spec = {
+        'name': 'user',
+        'scopes': ['self', 'shutdown'],
+    }
+    if role_users is not None:
+        role_spec['users'] = role_users
+    hub = MockHub(load_roles=[role_spec])
     hub.init_db()
-    hub.authenticator.allowed_users = user_names
+    db = hub.db
+    hub.authenticator.admin_users = set()
+    if allowed_users:
+        hub.authenticator.allowed_users = allowed_users
     await hub.init_role_creation()
-    await hub.init_users()
-    await hub.init_role_assignment()
-    user_role = orm.Role.find(hub.db, 'user')
-    for user_name in user_names:
-        user = orm.User.find(hub.db, user_name)
-        assert user in user_role.users
 
 
 async def test_user_config_creates_default_role():
@@ -1243,16 +1255,45 @@ async def test_admin_role_respects_config():
         assert user in admin_role.users
 
 
-async def test_empty_admin_spec():
-    role_spec = [{'name': 'admin', 'users': []}]
-    hub = MockHub(load_roles=role_spec)
+@pytest.mark.parametrize(
+    "in_db, role_users, admin_users, expected_members",
+    [
+        # users in the db, not specified in custom user role
+        # no change to membership
+        (["alpha", "beta"], None, None, ["alpha", "beta"]),
+        # admin_users is additive, not strict
+        (["alpha", "beta"], None, {"gamma"}, ["alpha", "beta", "gamma"]),
+        # explicit empty revokes all assignments
+        (["alpha", "beta"], [], None, []),
+        # explicit value is respected exactly
+        (["alpha", "beta"], ["alpha", "gamma"], None, ["alpha", "gamma"]),
+    ],
+)
+async def test_admin_role_membership(in_db, role_users, admin_users, expected_members):
+
+    load_roles = []
+    if role_users is not None:
+        load_roles.append({"name": "admin", "users": role_users})
+    if not admin_users:
+        admin_users = set()
+    hub = MockHub(load_roles=load_roles, db_url="sqlite:///:memory:")
     hub.init_db()
-    hub.authenticator.admin_users = []
     await hub.init_role_creation()
+    db = hub.db
+    hub.authenticator.admin_users = admin_users
+    # add in_db users to the database
+    # this is the 'before' state of who had the role before startup
+    for username in in_db or []:
+        user = orm.User(name=username)
+        db.add(user)
+        db.commit()
+        roles.grant_role(db, user, "admin")
+    db.commit()
     await hub.init_users()
     await hub.init_role_assignment()
-    admin_role = orm.Role.find(hub.db, 'admin')
-    assert not admin_role.users
+    admin_role = orm.Role.find(db, 'admin')
+    role_members = sorted(user.name for user in admin_role.users)
+    assert role_members == expected_members
 
 
 async def test_no_default_service_role():