Repository: jupyterhub/jupyterhub

Issue:
Ideas for new authenticator config
I agreed with @minrk in a the recent June jupyterhub team meeting to describe an idea of new config without coupling it to suggesting how it should be implemented.

I want directly configured behaviors instead of behavior following from if `allowed_users` is truthy or falsy. Currently the following behaviors can follow from declaring `allowed_users`:

1. Declaring `allowed_users` truthy currently [implies allowing existing users](https://github.com/jupyterhub/jupyterhub/blob/4.0.0/jupyterhub/auth.py#L647-L648) to login as well.
2. Declaring `allowed_users` falsy can for some authenticators in certain configurations imply allowing all authenticated users to login.

I suggest new authentication config like below to provide a way to directly configure the otherwise implied behavior:

1. `allow_existing_users`
2. `allow_all`

I think such new config could make:
- a deployments jupyterhub config easier to understand without reading the docs
- easier to provide as secure by default
- easier to document, maintain, and implement

## Related

- Discussion related to `allow_existing_users`: #4483
- Discussion related to `allow_all`: #4484
- Discussion related to defaulting to not allowing existing users in OAuthenticator, introducing `allow_existing_users` and defaulting to False: https://github.com/jupyterhub/oauthenticator/issues/619
- Discussion related to defaulting to not allowing all users in OAuthenticator when no users is explicitly allowed, introducing `allow_all` and defaulting to False: https://github.com/jupyterhub/oauthenticator/issues/620


Discussion:
Is this a separate issue? It seems to be a summary of the two already existing issues and there are no additional actions to take or separate conversation to have. Do you want to close the other two and merge the discussion here?
I opened this understanding that you wanted to take a step back and think about what we want to accomplish and how from the perspective of what config to provide, separated from technically how which the other issues had started considering.

This is highly coupled to all issues linked under the Related heading, and I'm not sure how to organize the discussion. =/
I think your proposal is solid, and we have a good path forward on the issues you've linked, individually. Maybe here is a good place to try to get a high-level overview of the use cases we want to make sure work, and what their config looks like, so we have something that won't get auto-closed by PRs addressing individual issues. I think it's appropriate to start at a high level and say "I want these user login/creation patterns to work, and I think the config should look like this" ignoring (at first) how everything actually works. Then we can try to fill the gap between what we have and what we want.

For example:

- fully deferred configuration to an Authenticator's authentication mechanism (e.g. PAM, LDAP, default behavior)
- users managed via the API (e.g. most target uses of nativeauthenticator)
- always-config-specified users and groups (revoking membership revokes permission - `allow_existing_users`)
- documenting that allowed_users when combined with other allow mechanisms should be a _union_ instead of an intersection (in-progress implementation change in oauthenticator, can be part of the 'spec' in base Authenticator, though implementation is still up to subclasses). Maybe some API changes in the base class can make this easier to implement in a consistent way.
- pre-populating a user list while also allowing future logins

Further related questions, that might help (or might not):

- do we need to more explicitly separate authentication from authorization? We have historically conflated these two, but we do now have _granular_ authorization in role membership. What there isn't currently (for the most part) is a separation between an _authenticated_ user and membership in the `user` role, despite the fact that it's technically possible. There aren't easy ways to make it happen.

There's an open PR adding some authorisation support (user/admin/none) for Azure: https://github.com/jupyterhub/oauthenticator/pull/446

So there's clearly demand for it. I think we should come up with a position regarding authorisation, even if it's that we can't support it, to provide clarity to developers.
We do already have support for logging in and being granted a variety of permission levels (JupyterHub supports it, authenticators must implement the mapping within authenticate and can return things like group membership). I don't yet see any demand for allowing users to complete authentication _at the jupyterhub level_ without granting at least `user`-level permission.

Resolving patch:
diff --git a/jupyterhub/app.py b/jupyterhub/app.py
--- a/jupyterhub/app.py
+++ b/jupyterhub/app.py
@@ -2077,6 +2077,9 @@ async def init_users(self):
                     "auth_state is enabled, but encryption is not available: %s" % e
                 )
 
+        # give the authenticator a chance to check its own config
+        self.authenticator.check_allow_config()
+
         if self.admin_users and not self.authenticator.admin_users:
             self.log.warning(
                 "\nJupyterHub.admin_users is deprecated since version 0.7.2."
@@ -2104,9 +2107,9 @@ async def init_users(self):
                 new_users.append(user)
             else:
                 user.admin = True
+
         # the admin_users config variable will never be used after this point.
         # only the database values will be referenced.
-
         allowed_users = [
             self.authenticator.normalize_username(name)
             for name in self.authenticator.allowed_users
@@ -2116,10 +2119,10 @@ async def init_users(self):
             if not self.authenticator.validate_username(username):
                 raise ValueError("username %r is not valid" % username)
 
-        if not allowed_users:
-            self.log.info(
-                "Not using allowed_users. Any authenticated user will be allowed."
-            )
+        if self.authenticator.allowed_users and self.authenticator.admin_users:
+            # make sure admin users are in the allowed_users set, if defined,
+            # otherwise they won't be able to login
+            self.authenticator.allowed_users |= self.authenticator.admin_users
 
         # add allowed users to the db
         for name in allowed_users:
diff --git a/jupyterhub/auth.py b/jupyterhub/auth.py
--- a/jupyterhub/auth.py
+++ b/jupyterhub/auth.py
@@ -121,6 +121,55 @@ def _deprecated_db(self):
         """
     ).tag(config=True)
 
+    any_allow_config = Bool(
+        False,
+        help="""Is there any allow config?
+        
+        Used to show a warning if it looks like nobody can access the Hub,
+        which can happen when upgrading to JupyterHub 5,
+        now that `allow_all` defaults to False.
+        
+        Deployments can set this explicitly to True to suppress
+        the "No allow config found" warning.
+        
+        Will be True if any config tagged with `.tag(allow_config=True)`
+        or starts with `allow` is truthy.
+        
+        .. versionadded:: 5.0
+        """,
+    ).tag(config=True)
+
+    @default("any_allow_config")
+    def _default_any_allowed(self):
+        for trait_name, trait in self.traits(config=True).items():
+            if trait.metadata.get("allow_config", False) or trait_name.startswith(
+                "allow"
+            ):
+                # this is only used for a helpful warning, so not the biggest deal if it's imperfect
+                if getattr(self, trait_name):
+                    return True
+        return False
+
+    def check_allow_config(self):
+        """Log a warning if no allow config can be found.
+
+        Could get a false positive if _only_ unrecognized allow config is used.
+        Authenticators can apply `.tag(allow_config=True)` to label this config
+        to make sure it is found.
+
+        Subclasses can override to perform additonal checks and warn about likely
+        authenticator configuration problems.
+
+        .. versionadded:: 5.0
+        """
+        if not self.any_allow_config:
+            self.log.warning(
+                "No allow config found, it's possible that nobody can login to your Hub!\n"
+                "You can set `c.Authenticator.allow_all = True` to allow any user who can login to access the Hub,\n"
+                "or e.g. `allowed_users` to a set of users who should have access.\n"
+                "You may suppress this warning by setting c.Authenticator.any_allow_config = True."
+            )
+
     whitelist = Set(
         help="Deprecated, use `Authenticator.allowed_users`",
         config=True,
@@ -132,7 +181,7 @@ def _deprecated_db(self):
 
         Use this to limit which authenticated users may login.
         Default behavior: only users in this set are allowed.
-        
+
         If empty, does not perform any restriction,
         in which case any authenticated user is allowed.
 
@@ -144,6 +193,83 @@ def _deprecated_db(self):
         """
     ).tag(config=True)
 
+    allow_all = Bool(
+        False,
+        config=True,
+        help="""
+        Allow every user who can successfully authenticate to access JupyterHub.
+        
+        False by default, which means for most Authenticators,
+        _some_ allow-related configuration is required to allow users to log in.
+
+        Authenticator subclasses may override the default with e.g.::
+
+            @default("allow_all")
+            def _default_allow_all(self):
+                # if _any_ auth config (depends on the Authenticator)
+                if self.allowed_users or self.allowed_groups or self.allow_existing_users:
+                    return False
+                else:
+                    return True
+
+        .. versionadded:: 5.0
+        
+        .. versionchanged:: 5.0
+            Prior to 5.0, `allow_all` wasn't defined on its own,
+            and was instead implicitly True when no allow config was provided,
+            i.e. `allowed_users` unspecified or empty on the base Authenticator class.
+            
+            To preserve pre-5.0 behavior,
+            set `allow_all = True` if you have no other allow configuration.
+        """,
+    ).tag(allow_config=True)
+
+    allow_existing_users = Bool(
+        # dynamic default computed from allowed_users
+        config=True,
+        help="""
+        Allow existing users to login.
+
+        Defaults to True if `allowed_users` is set for historical reasons, and
+        False otherwise.
+
+        With this enabled, all users present in the JupyterHub database are allowed to login.
+        This has the effect of any user who has _previously_ been allowed to login
+        via any means will continue to be allowed until the user is deleted via the /hub/admin page
+        or REST API.
+
+        .. warning::
+
+           Before enabling this you should review the existing users in the
+           JupyterHub admin panel at `/hub/admin`. You may find users existing
+           there because they have previously been declared in config such as
+           `allowed_users` or allowed to sign in.
+
+        .. warning::
+
+           When this is enabled and you wish to remove access for one or more
+           users previously allowed, you must make sure that they
+           are removed from the jupyterhub database. This can be tricky to do
+           if you stop allowing an externally managed group of users for example.
+
+        With this enabled, JupyterHub admin users can visit `/hub/admin` or use
+        JupyterHub's REST API to add and remove users to manage who can login.
+
+        .. versionadded:: 5.0
+        """,
+    ).tag(allow_config=True)
+
+    @default("allow_existing_users")
+    def _allow_existing_users_default(self):
+        """
+        Computes the default value of allow_existing_users based on if
+        allowed_users to align with original behavior not introduce a breaking
+        change.
+        """
+        if self.allowed_users:
+            return True
+        return False
+
     blocked_users = Set(
         help="""
         Set of usernames that are not allowed to log in.
@@ -472,8 +598,7 @@ def check_allowed(self, username, authentication=None):
             web.HTTPError(403):
                 Raising HTTPErrors directly allows customizing the message shown to the user.
         """
-        if not self.allowed_users:
-            # No allowed set means any name is allowed
+        if self.allow_all:
             return True
         return username in self.allowed_users
 
@@ -525,8 +650,9 @@ async def get_authenticated_user(self, handler, data):
         The various stages can be overridden separately:
          - `authenticate` turns formdata into a username
          - `normalize_username` normalizes the username
-         - `check_allowed` checks against the allowed usernames
          - `check_blocked_users` check against the blocked usernames
+         - `allow_all` is checked
+         - `check_allowed` checks against the allowed usernames
          - `is_admin` check if a user is an admin
 
         .. versionchanged:: 0.8
@@ -560,7 +686,11 @@ async def get_authenticated_user(self, handler, data):
             self.log.warning("User %r blocked. Stop authentication", username)
             return
 
-        allowed_pass = await maybe_future(self.check_allowed(username, authenticated))
+        allowed_pass = self.allow_all
+        if not allowed_pass:
+            allowed_pass = await maybe_future(
+                self.check_allowed(username, authenticated)
+            )
 
         if allowed_pass:
             if authenticated['admin'] is None:
@@ -677,25 +807,31 @@ def add_user(self, user):
         """Hook called when a user is added to JupyterHub
 
         This is called:
-         - When a user first authenticates
-         - When the hub restarts, for all users.
+         - When a user first authenticates, _after_ all allow and block checks have passed
+         - When the hub restarts, for all users in the database (i.e. users previously allowed)
+         - When a user is added to the database, either via configuration or REST API
 
         This method may be a coroutine.
 
-        By default, this just adds the user to the allowed_users set.
+        By default, this adds the user to the allowed_users set if
+        allow_existing_users is true.
 
-        Subclasses may do more extensive things, such as adding actual unix users,
+        Subclasses may do more extensive things, such as creating actual system users,
         but they should call super to ensure the allowed_users set is updated.
 
         Note that this should be idempotent, since it is called whenever the hub restarts
         for all users.
 
+        .. versionchanged:: 5.0
+           Now adds users to the allowed_users set if allow_all is False and allow_existing_users is True,
+           instead of if allowed_users is not empty.
+
         Args:
             user (User): The User wrapper object
         """
         if not self.validate_username(user.name):
             raise ValueError("Invalid username: %s" % user.name)
-        if self.allowed_users:
+        if self.allow_existing_users and not self.allow_all:
             self.allowed_users.add(user.name)
 
     def delete_user(self, user):
@@ -902,23 +1038,16 @@ def _add_user_cmd_default(self):
         help="""
         Allow login from all users in these UNIX groups.
 
-        If set, allowed username set is ignored.
+        .. versionchanged:: 5.0
+            `allowed_groups` may be specified together with allowed_users,
+            to grant access by group OR name.
         """
-    ).tag(config=True)
-
-    @observe('allowed_groups')
-    def _allowed_groups_changed(self, change):
-        """Log a warning if mutually exclusive user and group allowed sets are specified."""
-        if self.allowed_users:
-            self.log.warning(
-                "Ignoring Authenticator.allowed_users set because Authenticator.allowed_groups supplied!"
-            )
+    ).tag(config=True, allow_config=True)
 
     def check_allowed(self, username, authentication=None):
-        if self.allowed_groups:
-            return self.check_allowed_groups(username, authentication)
-        else:
-            return super().check_allowed(username, authentication)
+        if self.check_allowed_groups(username, authentication):
+            return True
+        return super().check_allowed(username, authentication)
 
     def check_allowed_groups(self, username, authentication=None):
         """
@@ -1248,8 +1377,20 @@ class DummyAuthenticator(Authenticator):
     if it logs in with that password.
 
     .. versionadded:: 1.0
+
+    .. versionadded:: 5.0
+        `allow_all` defaults to True,
+        preserving default behavior.
     """
 
+    @default("allow_all")
+    def _allow_all_default(self):
+        if self.allowed_users:
+            return False
+        else:
+            # allow all by default
+            return True
+
     password = Unicode(
         config=True,
         help="""
@@ -1259,6 +1400,12 @@ class DummyAuthenticator(Authenticator):
         """,
     )
 
+    def check_allow_config(self):
+        super().check_allow_config()
+        self.log.warning(
+            f"Using testing authenticator {self.__class__.__name__}! This is not meant for production!"
+        )
+
     async def authenticate(self, handler, data):
         """Checks against a global password if it's been set. If not, allow any user/pass combo"""
         if self.password:


Test patch:
diff --git a/jupyterhub/tests/mocking.py b/jupyterhub/tests/mocking.py
--- a/jupyterhub/tests/mocking.py
+++ b/jupyterhub/tests/mocking.py
@@ -243,6 +243,8 @@ def __init__(self, *args, **kwargs):
             cert_location = kwargs['internal_certs_location']
             kwargs['external_certs'] = ssl_setup(cert_location, 'hub-ca')
         super().__init__(*args, **kwargs)
+        if 'allow_all' not in self.config.Authenticator:
+            self.config.Authenticator.allow_all = True
 
     @default('subdomain_host')
     def _subdomain_host_default(self):
diff --git a/jupyterhub/tests/test_app.py b/jupyterhub/tests/test_app.py
--- a/jupyterhub/tests/test_app.py
+++ b/jupyterhub/tests/test_app.py
@@ -475,6 +475,7 @@ async def test_user_creation(tmpdir, request):
     ]
 
     cfg = Config()
+    cfg.Authenticator.allow_all = False
     cfg.Authenticator.allowed_users = allowed_users
     cfg.JupyterHub.load_groups = groups
     cfg.JupyterHub.load_roles = roles
diff --git a/jupyterhub/tests/test_auth.py b/jupyterhub/tests/test_auth.py
--- a/jupyterhub/tests/test_auth.py
+++ b/jupyterhub/tests/test_auth.py
@@ -3,12 +3,13 @@
 # Copyright (c) Jupyter Development Team.
 # Distributed under the terms of the Modified BSD License.
 import logging
+from itertools import chain
 from unittest import mock
 from urllib.parse import urlparse
 
 import pytest
 from requests import HTTPError
-from traitlets import Any
+from traitlets import Any, Tuple
 from traitlets.config import Config
 
 from jupyterhub import auth, crypto, orm
@@ -18,7 +19,7 @@
 
 
 async def test_pam_auth():
-    authenticator = MockPAMAuthenticator()
+    authenticator = MockPAMAuthenticator(allow_all=True)
     authorized = await authenticator.get_authenticated_user(
         None, {'username': 'match', 'password': 'match'}
     )
@@ -37,7 +38,7 @@ async def test_pam_auth():
 
 
 async def test_pam_auth_account_check_disabled():
-    authenticator = MockPAMAuthenticator(check_account=False)
+    authenticator = MockPAMAuthenticator(allow_all=True, check_account=False)
     authorized = await authenticator.get_authenticated_user(
         None, {'username': 'allowedmatch', 'password': 'allowedmatch'}
     )
@@ -82,7 +83,9 @@ def getgrouplist(name, group):
         return user_group_map[name]
 
     authenticator = MockPAMAuthenticator(
-        admin_groups={'jh_admins', 'wheel'}, admin_users={'override_admin'}
+        admin_groups={'jh_admins', 'wheel'},
+        admin_users={'override_admin'},
+        allow_all=True,
     )
 
     # Check admin_group applies as expected
@@ -141,7 +144,10 @@ def getgrouplist(name, group):
 
 
 async def test_pam_auth_allowed():
-    authenticator = MockPAMAuthenticator(allowed_users={'wash', 'kaylee'})
+    authenticator = MockPAMAuthenticator(
+        allowed_users={'wash', 'kaylee'}, allow_all=False
+    )
+
     authorized = await authenticator.get_authenticated_user(
         None, {'username': 'kaylee', 'password': 'kaylee'}
     )
@@ -162,7 +168,7 @@ async def test_pam_auth_allowed_groups():
     def getgrnam(name):
         return MockStructGroup('grp', ['kaylee'])
 
-    authenticator = MockPAMAuthenticator(allowed_groups={'group'})
+    authenticator = MockPAMAuthenticator(allowed_groups={'group'}, allow_all=False)
 
     with mock.patch.object(authenticator, '_getgrnam', getgrnam):
         authorized = await authenticator.get_authenticated_user(
@@ -179,14 +185,14 @@ def getgrnam(name):
 
 async def test_pam_auth_blocked():
     # Null case compared to next case
-    authenticator = MockPAMAuthenticator()
+    authenticator = MockPAMAuthenticator(allow_all=True)
     authorized = await authenticator.get_authenticated_user(
         None, {'username': 'wash', 'password': 'wash'}
     )
     assert authorized['name'] == 'wash'
 
-    # Blacklist basics
-    authenticator = MockPAMAuthenticator(blocked_users={'wash'})
+    # Blocklist basics
+    authenticator = MockPAMAuthenticator(blocked_users={'wash'}, allow_all=True)
     authorized = await authenticator.get_authenticated_user(
         None, {'username': 'wash', 'password': 'wash'}
     )
@@ -194,7 +200,9 @@ async def test_pam_auth_blocked():
 
     # User in both allowed and blocked: default deny.  Make error someday?
     authenticator = MockPAMAuthenticator(
-        blocked_users={'wash'}, allowed_users={'wash', 'kaylee'}
+        blocked_users={'wash'},
+        allowed_users={'wash', 'kaylee'},
+        allow_all=True,
     )
     authorized = await authenticator.get_authenticated_user(
         None, {'username': 'wash', 'password': 'wash'}
@@ -203,7 +211,8 @@ async def test_pam_auth_blocked():
 
     # User not in blocked set can log in
     authenticator = MockPAMAuthenticator(
-        blocked_users={'wash'}, allowed_users={'wash', 'kaylee'}
+        blocked_users={'wash'},
+        allowed_users={'wash', 'kaylee'},
     )
     authorized = await authenticator.get_authenticated_user(
         None, {'username': 'kaylee', 'password': 'kaylee'}
@@ -221,7 +230,8 @@ async def test_pam_auth_blocked():
 
     # User in neither list
     authenticator = MockPAMAuthenticator(
-        blocked_users={'mal'}, allowed_users={'wash', 'kaylee'}
+        blocked_users={'mal'},
+        allowed_users={'wash', 'kaylee'},
     )
     authorized = await authenticator.get_authenticated_user(
         None, {'username': 'simon', 'password': 'simon'}
@@ -257,7 +267,9 @@ def deprecated_xlist(self, username):
 
 
 async def test_pam_auth_no_such_group():
-    authenticator = MockPAMAuthenticator(allowed_groups={'nosuchcrazygroup'})
+    authenticator = MockPAMAuthenticator(
+        allowed_groups={'nosuchcrazygroup'},
+    )
     authorized = await authenticator.get_authenticated_user(
         None, {'username': 'kaylee', 'password': 'kaylee'}
     )
@@ -405,7 +417,7 @@ async def test_auth_state_disabled(app, auth_state_unavailable):
 
 
 async def test_normalize_names():
-    a = MockPAMAuthenticator()
+    a = MockPAMAuthenticator(allow_all=True)
     authorized = await a.get_authenticated_user(
         None, {'username': 'ZOE', 'password': 'ZOE'}
     )
@@ -428,7 +440,7 @@ async def test_normalize_names():
 
 
 async def test_username_map():
-    a = MockPAMAuthenticator(username_map={'wash': 'alpha'})
+    a = MockPAMAuthenticator(username_map={'wash': 'alpha'}, allow_all=True)
     authorized = await a.get_authenticated_user(
         None, {'username': 'WASH', 'password': 'WASH'}
     )
@@ -458,7 +470,7 @@ def test_auth_hook(authenticator, handler, authentication):
         authentication['testkey'] = 'testvalue'
         return authentication
 
-    a = MockPAMAuthenticator(post_auth_hook=test_auth_hook)
+    a = MockPAMAuthenticator(allow_all=True, post_auth_hook=test_auth_hook)
 
     authorized = await a.get_authenticated_user(
         None, {'username': 'test_user', 'password': 'test_user'}
@@ -566,6 +578,7 @@ async def test_auth_managed_groups(
         parent=app,
         authenticated_groups=authenticated_groups,
         refresh_groups=refresh_groups,
+        allow_all=True,
     )
 
     user.groups.append(group)
@@ -593,3 +606,193 @@ async def test_auth_managed_groups(
         assert not app.db.dirty
         groups = sorted(g.name for g in user.groups)
         assert groups == expected_refresh_groups
+
+
+@pytest.mark.parametrize(
+    "allowed_users,  allow_existing_users",
+    [
+        ('specified', True),
+        ('', False),
+    ],
+)
+async def test_allow_defaults(app, user, allowed_users, allow_existing_users):
+    if allowed_users:
+        allowed_users = set(allowed_users.split(','))
+    else:
+        allowed_users = set()
+    authenticator = auth.Authenticator(allowed_users=allowed_users)
+    authenticator.authenticate = lambda handler, data: data["username"]
+    assert authenticator.allow_all is False
+    assert authenticator.allow_existing_users == allow_existing_users
+
+    # user was already in the database
+    # this happens during hub startup
+    authenticator.add_user(user)
+    if allowed_users:
+        assert user.name in authenticator.allowed_users
+    else:
+        authenticator.allowed_users == set()
+
+    specified_allowed = await authenticator.get_authenticated_user(
+        None, {"username": "specified"}
+    )
+    if "specified" in allowed_users:
+        assert specified_allowed is not None
+    else:
+        assert specified_allowed is None
+
+    existing_allowed = await authenticator.get_authenticated_user(
+        None, {"username": user.name}
+    )
+    if allow_existing_users:
+        assert existing_allowed is not None
+    else:
+        assert existing_allowed is None
+
+
+@pytest.mark.parametrize("allow_all", [None, True, False])
+@pytest.mark.parametrize("allow_existing_users", [None, True, False])
+@pytest.mark.parametrize("allowed_users", ["existing", ""])
+def test_allow_existing_users(
+    app, user, allowed_users, allow_all, allow_existing_users
+):
+    if allowed_users:
+        allowed_users = set(allowed_users.split(','))
+    else:
+        allowed_users = set()
+    authenticator = auth.Authenticator(
+        allowed_users=allowed_users,
+    )
+    if allow_all is None:
+        # default allow_all
+        allow_all = authenticator.allow_all
+    else:
+        authenticator.allow_all = allow_all
+    if allow_existing_users is None:
+        # default allow_all
+        allow_existing_users = authenticator.allow_existing_users
+    else:
+        authenticator.allow_existing_users = allow_existing_users
+
+    # first, nobody in the database
+    assert authenticator.check_allowed("newuser") == allow_all
+
+    # user was already in the database
+    # this happens during hub startup
+    authenticator.add_user(user)
+    if allow_existing_users or allow_all:
+        assert authenticator.check_allowed(user.name)
+    else:
+        assert not authenticator.check_allowed(user.name)
+    for username in allowed_users:
+        assert authenticator.check_allowed(username)
+
+    assert authenticator.check_allowed("newuser") == allow_all
+
+
+@pytest.mark.parametrize("allow_all", [True, False])
+@pytest.mark.parametrize("allow_existing_users", [True, False])
+def test_allow_existing_users_first_time(user, allow_all, allow_existing_users):
+    # make sure that calling add_user doesn't change results
+    authenticator = auth.Authenticator(
+        allow_all=allow_all,
+        allow_existing_users=allow_existing_users,
+    )
+    allowed_before_one = authenticator.check_allowed(user.name)
+    allowed_before_two = authenticator.check_allowed("newuser")
+    # add_user is called after successful login
+    # it shouldn't change results (e.g. by switching .allowed_users from empty to non-empty)
+    if allowed_before_one:
+        authenticator.add_user(user)
+    assert authenticator.check_allowed(user.name) == allowed_before_one
+    assert authenticator.check_allowed("newuser") == allowed_before_two
+
+
+class AllowAllIgnoringAuthenticator(auth.Authenticator):
+    """Test authenticator with custom check_allowed
+
+    not updated for allow_all, allow_existing_users
+
+    Make sure new config doesn't break backward-compatibility
+    or grant unintended access for Authenticators written before JupyterHub 5.
+    """
+
+    allowed_letters = Tuple(config=True, help="Initial letters to allow")
+
+    def authenticate(self, handler, data):
+        return {"name": data["username"]}
+
+    def check_allowed(self, username, auth=None):
+        if not self.allowed_users and not self.allowed_letters:
+            # this subclass doesn't know about the JupyterHub 5 allow_all config
+            # no allow config, allow all!
+            return True
+        if self.allowed_users and username in self.allowed_users:
+            return True
+        if self.allowed_letters and username.startswith(self.allowed_letters):
+            return True
+        return False
+
+
+# allow_all is not recognized by Authenticator subclass
+# make sure it doesn't make anything more permissive, at least
+@pytest.mark.parametrize("allow_all", [True, False])
+@pytest.mark.parametrize(
+    "allowed_users, allowed_letters, allow_existing_users, allowed, not_allowed",
+    [
+        ("", "", None, "anyone,should-be,allowed,existing", ""),
+        ("", "a,b", None, "alice,bebe", "existing,other"),
+        ("", "a,b", False, "alice,bebe", "existing,other"),
+        ("", "a,b", True, "alice,bebe,existing", "other"),
+        ("specified", "a,b", None, "specified,alice,bebe,existing", "other"),
+        ("specified", "a,b", False, "specified,alice,bebe", "existing,other"),
+        ("specified", "a,b", True, "specified,alice,bebe,existing", "other"),
+    ],
+)
+async def test_authenticator_without_allow_all(
+    app,
+    allowed_users,
+    allowed_letters,
+    allow_existing_users,
+    allowed,
+    not_allowed,
+    allow_all,
+):
+    kwargs = {}
+    if allow_all is not None:
+        kwargs["allow_all"] = allow_all
+    if allow_existing_users is not None:
+        kwargs["allow_existing_users"] = allow_existing_users
+    if allowed_users:
+        kwargs["allowed_users"] = set(allowed_users.split(','))
+    if allowed_letters:
+        kwargs["allowed_letters"] = tuple(allowed_letters.split(','))
+
+    authenticator = AllowAllIgnoringAuthenticator(**kwargs)
+
+    # load one user from db
+    existing_user = add_user(app.db, app, name="existing")
+    authenticator.add_user(existing_user)
+
+    if allowed:
+        allowed = allowed.split(",")
+    if not_allowed:
+        not_allowed = not_allowed.split(",")
+
+    expected_allowed = sorted(allowed)
+    expected_not_allowed = sorted(not_allowed)
+    to_check = list(chain(expected_allowed, expected_not_allowed))
+    if allow_all:
+        expected_allowed = to_check
+        expected_not_allowed = []
+
+    are_allowed = []
+    are_not_allowed = []
+    for username in to_check:
+        if await authenticator.get_authenticated_user(None, {"username": username}):
+            are_allowed.append(username)
+        else:
+            are_not_allowed.append(username)
+
+    assert are_allowed == expected_allowed
+    assert are_not_allowed == expected_not_allowed