Repository: jupyterhub/zero-to-jupyterhub-k8s

Issue:
Hub existingSecret functionality not working as expected 
<!-- Thank you for contributing. These HTML comments will not render in the issue, but you can delete them once you've read them if you prefer! -->

### Bug description
<!-- Use this section to clearly and concisely describe the bug. -->
Hi,  I am trying to update jupyterhub to helm chart 0.10.6 from 0.9.0, and with the recent change of moving values.yaml only to hub-secret and removing it from hub-config is breaking the Hub existingSecret functionality. In our existing deployment on EKS we use gitops/flux/helm operator for our deployments and we didn't want to add secrets to git, so we created hub-secret which only includes secrets(proxy token, db creds, crypto-key, cookie-secret, etc) using [external secrets operator](https://github.com/external-secrets/kubernetes-external-secrets)  and set the name as existingSecret. Now when the existingSecret is set, the secret is not created and values.yaml doesn't make it to the hub pod and it doesn't start properly.


#### Expected behaviour
The previous behaviour of existingSecret should continue to work without having to add values.yaml to existingSecret. 






Discussion:
Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
<br>If you haven't done so already, check out [Jupyter's Code of Conduct](https://github.com/jupyter/governance/blob/master/conduct/code_of_conduct.md). Also, please try to follow the issue template as it helps other other community members to contribute more effectively.
![welcome](https://raw.githubusercontent.com/jupyterhub/.github/master/images/welcome.jpg)
You can meet the other [Jovyans](https://jupyter.readthedocs.io/en/latest/community/content-community.html?highlight=jovyan#what-is-a-jovyan) by joining our [Discourse forum](http://discourse.jupyter.org/). There is also an intro thread there where you can stop by and say Hi! :wave:
<br>Welcome to the Jupyter community! :tada:

Yeah =/ This is the kind of experience I feared from accepting this feature in the first place. The [documentation has](https://zero-to-jupyterhub.readthedocs.io/en/stable/resources/reference.html#hub-existingsecret) a note about you needing to keep it updated.

## Short term
Short term, what you are required to do is to specify your passwords etc without hub.existingSecret set, then re-create the k8s secret, and then set hub.existingSecret again and remove your specification of passwords etc.

## Long term
My understanding of the feature wanted is: to be able to extract all secret information from a helm configuration file, and letting the Helm chart pick up on those from a k8s Secret.

Currently this works, but, the major drawback is that one need to manually manage the k8s Secret even though the only change was in default configuration between Helm chart versions rather than the user provided passwords.

I think it is possible to implement a better solution. Such solution should improve the user experience but also not add too fragile logic that is hard to maintain.

### Brainstorming long term solution
- `proxy.secretToken` is special a special case because its used outside the hub pod, but we can manage this with #1993.
- We could make `hub.existingSecret` be an optionally mounted k8s secret to the hub pod, which then is only supposed to contain a `values.yaml` entry that we then load and merge into other values.

## Related
- Functionality added in #1142
- Requested in #1622
- Upcoming functionality to not need to specify proxy.secretToken / hub.cookieSecret / hub.config.CryptKeeper.keys in #1993
@minrk @manics @yuvipanda - what do you think about the brainstormed solution above to making `hub.existingSecret` a sustainable practice for users/maintainers?

Sounds to me like a potential 1.0.0 enhancement to aim for.
This issue has been mentioned on **Jupyter Community Forum**. There might be relevant details there:

https://discourse.jupyter.org/t/how-am-i-supposed-to-use-existingsecret/5097/4

# Reimplementation of hub.existingSecret

This is my preliminary strategy planned to make hub.existingSecret sustainable to use to manage sensitive configuration needed by the Helm chart.

Use of hub.existingSecret hasn't been a good experience for users because they end up needing to manually update it whenever they change something passed to the hub pod through values.yaml to be read in jupyterhub_config.py.

My idea is to solve this by allowing hub.existingSecret be mounted in parallel to the default k8s Secret resource we make use of and then merging values. For keys such as hub.db.password and hub.config.JupyterHub.proxy_auth_token that are root level keys in the secrets, the idea is to always favor reference to an hub.existingSecret if there is one set.

## Implementation challenges

1. JupyterHub.proxy_auth_token and hub.db.password are used from helm templates (hub / proxy deployment env vars). Even if we have a existingSecret, we can't inspect it. and must reference one or the other secret - there is no merge functionality we can use here as we don't actually probe the content but instead only declare where it resides. This forces those fields to be pinned in one or the other Secret. Due to this, we should always favor existingSecret content over defaultSecret content whenever it could be defined in either.

   Since we need to favor looking for hub.db.password and JupyterHub.proxy_auth_token in the existingSecret, usage of it will require it to be found there.

2. The Helm lookup function that can be used during template rendering is only able to lookup chart managed resources, so not the existingSecret even if we wanted to.

   This is causing a problem for the automatic generation of the proxy_auth_token secret because we can't lookup the existingSecret while also giving priority to existingSecret, and it would force the user to declare something there that otherwise can be autogenerated since #1993.

   Due to this, I suggest we lookup JupyterHub.proxy_auth_token in the defaultSecret at all time and write an exception notice regarding that in the docs of hub.existingSecret.

## Implementation steps

1. Let hub.existingSecret mount in parallel to the defaultSecret in the hub pod.
2. Merge in existingSecret values with defaultSecret values, and let top level  keys override defaultSecret values if they are defined. This can be handled in z2jh.py.
3. Create three separate _helpers-names.tpl templates and make use of them where appropriate.
   - `hub`                                used for: volume declaration hub pod, mounting env var from JupyterHub.proxy_auth_token
   - `hub-existing-secret`                used for: volume declaration hub pod (conditional)
   - `hub-existing-secret-or-default`     used for: mounting of hub.db.password env var
4. Update configuration reference
5. changelog strategy: point to docs

---

Feedback on the strategy appreciated of course.

Resolving patch:
diff --git a/jupyterhub/files/hub/jupyterhub_config.py b/jupyterhub/files/hub/jupyterhub_config.py
--- a/jupyterhub/files/hub/jupyterhub_config.py
+++ b/jupyterhub/files/hub/jupyterhub_config.py
@@ -66,6 +66,12 @@ def camelCaseify(s):
     c.JupyterHub.db_url = "sqlite://"
 else:
     set_config_if_not_none(c.JupyterHub, "db_url", "hub.db.url")
+db_password = get_secret_value("hub.db.password", None)
+if db_password is not None:
+    if db_type == "mysql":
+        os.environ["MYSQL_PWD"] = db_password
+    elif db_type == "postgres":
+        os.environ["PGPASSWORD"] = db_password
 
 
 # c.JupyterHub configuration from Helm chart's configmap
@@ -360,13 +366,17 @@ def camelCaseify(s):
     )
 
 for name, service in get_config("hub.services", {}).items():
-    # jupyterhub.services is a list of dicts, but
-    # in the helm chart it is a dict of dicts for easier merged-config
+    # c.JupyterHub.services is a list of dicts, but
+    # hub.services is a dict of dicts to make the config mergable
     service.setdefault("name", name)
-    # handle camelCase->snake_case of api_token
-    api_token = service.pop("apiToken", None)
+
+    # As the api_token could be exposed in hub.existingSecret, we need to read
+    # it it from there or fall back to the chart managed k8s Secret's value.
+    service.pop("apiToken", None)
+    api_token = get_secret_value(f"hub.services.{service['name']}.apiToken", None)
     if api_token:
         service["api_token"] = api_token
+
     c.JupyterHub.services.append(service)
 
 
@@ -420,18 +430,25 @@ def camelCaseify(s):
         exec(compile(source=file_content, filename=file_name, mode="exec"))
 
 # load potentially seeded secrets
-c.JupyterHub.proxy_auth_token = get_secret_value("JupyterHub.proxy_auth_token")
-c.JupyterHub.cookie_secret = a2b_hex(get_secret_value("JupyterHub.cookie_secret"))
-c.CryptKeeper.keys = get_secret_value("CryptKeeper.keys").split(";")
-
-# load hub.config values, except potentially seeded secrets
-for section, sub_cfg in get_config("hub.config", {}).items():
-    if section == "JupyterHub" and sub_cfg in ["proxy_auth_token", "cookie_secret"]:
-        pass
-    elif section == "CryptKeeper" and sub_cfg in ["keys"]:
-        pass
-    else:
-        c[section].update(sub_cfg)
+#
+# NOTE: ConfigurableHTTPProxy.auth_token is set through an environment variable
+#       that is set using the chart managed secret.
+c.JupyterHub.cookie_secret = a2b_hex(
+    get_secret_value("hub.config.JupyterHub.cookie_secret")
+)
+c.CryptKeeper.keys = get_secret_value("hub.config.CryptKeeper.keys").split(";")
+
+# load hub.config values, except potentially seeded secrets already loaded
+for app, cfg in get_config("hub.config", {}).items():
+    if app == "JupyterHub":
+        cfg.pop("proxy_auth_token", None)
+        cfg.pop("cookie_secret", None)
+        cfg.pop("services", None)
+    elif app == "ConfigurableHTTPProxy":
+        cfg.pop("auth_token", None)
+    elif app == "CryptKeeper":
+        cfg.pop("keys", None)
+    c[app].update(cfg)
 
 # execute hub.extraConfig string
 extra_config = get_config("hub.extraConfig", {})
diff --git a/jupyterhub/files/hub/z2jh.py b/jupyterhub/files/hub/z2jh.py
--- a/jupyterhub/files/hub/z2jh.py
+++ b/jupyterhub/files/hub/z2jh.py
@@ -13,15 +13,20 @@
 @lru_cache()
 def _load_config():
     """Load the Helm chart configuration used to render the Helm templates of
-    the chart from a mounted k8s Secret."""
-
-    path = f"/usr/local/etc/jupyterhub/secret/values.yaml"
-    if os.path.exists(path):
-        print(f"Loading {path}")
-        with open(path) as f:
-            return yaml.safe_load(f)
-    else:
-        raise Exception(f"{path} not found!")
+    the chart from a mounted k8s Secret, and merge in values from an optionally
+    mounted secret (hub.existingSecret)."""
+
+    cfg = {}
+    for source in ("secret/values.yaml", "existing-secret/values.yaml"):
+        path = f"/usr/local/etc/jupyterhub/{source}"
+        if os.path.exists(path):
+            print(f"Loading {path}")
+            with open(path) as f:
+                values = yaml.safe_load(f)
+            cfg = _merge_dictionaries(cfg, values)
+        else:
+            print(f"No config at {path}")
+    return cfg
 
 
 @lru_cache()
@@ -37,15 +42,18 @@ def _get_config_value(key):
 
 
 @lru_cache()
-def get_secret_value(key):
-    """Load value from the k8s Secret given a key."""
-
-    path = f"/usr/local/etc/jupyterhub/secret/{key}"
-    if os.path.exists(path):
-        with open(path) as f:
-            return f.read()
-    else:
-        raise Exception(f"{path} not found!")
+def get_secret_value(key, default="never-explicitly-set"):
+    """Load value from the user managed k8s Secret or the default k8s Secret
+    given a key."""
+
+    for source in ("existing-secret", "secret"):
+        path = f"/usr/local/etc/jupyterhub/{source}/{key}"
+        if os.path.exists(path):
+            with open(path) as f:
+                return f.read()
+    if default != "never-explicitly-set":
+        return default
+    raise Exception(f"{key} not found in either k8s Secret!")
 
 
 def get_name(name):


Test patch:
diff --git a/tests/conftest.py b/tests/conftest.py
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -5,7 +5,9 @@
 
 import os
 import requests
+import textwrap
 import uuid
+
 from urllib.parse import urlparse
 
 import pytest
@@ -121,3 +123,21 @@ def jupyter_user(api_request):
     yield username
     r = api_request.delete("/users/" + username)
     assert r.status_code == 204
+
+
+@pytest.fixture(scope="module")
+def extra_files_test_command():
+    return textwrap.dedent(
+        """
+        ls -l /tmp/binaryData.txt | grep -- -rw-rw-rw- || exit 1
+        ls -l /tmp/dir1/binaryData.txt | grep -- -rw-rw-rw- || exit 2
+        ls -l /tmp/stringData.txt | grep -- -rw-rw-rw- || exit 3
+        ls -l /tmp/dir1/stringData.txt | grep -- -rw-rw-rw- || exit 4
+        ls -l /etc/test/data.yaml | grep -- -r--r--r-- || exit 5
+        ls -l /etc/test/data.yml | grep -- -r--r--r-- || exit 6
+        ls -l /etc/test/data.json | grep -- -r--r--r-- || exit 7
+        ls -l /etc/test/data.toml | grep -- -r--r--r-- || exit 8
+        cat /tmp/binaryData.txt | grep -- "hello world" || exit 9
+        cat /tmp/stringData.txt | grep -- "hello world" || exit 10
+        """
+    ).strip()
diff --git a/tests/test_hub.py b/tests/test_hub.py
new file mode 100644
--- /dev/null
+++ b/tests/test_hub.py
@@ -0,0 +1,243 @@
+"""
+These tests commonl use JupyterHub's REST API:
+http://petstore.swagger.io/?url=https://raw.githubusercontent.com/jupyterhub/jupyterhub/master/docs/rest-api.yml
+"""
+
+import os
+import re
+import subprocess
+import time
+
+import pytest
+import requests
+import yaml
+
+
+def test_api_root(api_request):
+    """
+    Tests the hub api's root endpoint (/). The hub's version should be returned.
+
+    A typical jupyterhub logging response to this test:
+
+        [I 2019-09-25 12:03:12.051 JupyterHub log:174] 200 GET /hub/api (test@127.0.0.1) 9.57ms
+    """
+
+    # load app version of chart
+    here = os.path.dirname(os.path.abspath(__file__))
+    chart_yaml = os.path.join(here, os.pardir, "jupyterhub", "Chart.yaml")
+
+    with open(chart_yaml) as f:
+        chart = yaml.safe_load(f)
+        jupyterhub_version = chart["appVersion"]
+
+    print("asking for the hub's version")
+    r = api_request.get("")
+    assert r.status_code == 200
+    assert r.json().get("version", "version-missing") == jupyterhub_version
+
+
+def test_api_info(api_request):
+    """
+    Tests the hub api's /info endpoint. Information about the hub should be
+    returned.
+
+    A typical jupyterhub logging response to this test:
+
+        [I 2019-09-25 12:03:12.086 JupyterHub log:174] 200 GET /hub/api/info (test@127.0.0.1) 10.21ms
+    """
+
+    print("asking for the hub information")
+    r = api_request.get("/info")
+    assert r.status_code == 200
+    result = r.json()
+    assert result["spawner"]["class"] == "kubespawner.spawner.KubeSpawner"
+
+
+def test_api_create_and_get_user(api_request, jupyter_user):
+    """
+    Tests the hub api's /users/:user endpoint, both POST and GET.
+
+    A jupyter user is created and commited to the hub database through the
+    jupyter_user pytest fixture declared in conftest.py. Due to this, this first
+    test to use the jupyter_user fixture is actually testing both the pytest
+    fixture that creates the user as well as the ability to get information from
+    the hub about the user. The jupyter_user fixture will automatically clean up
+    the user from the hub's database when this function exits.
+
+    A typical jupyterhub logging response to this test:
+
+        [I 2019-09-25 12:03:12.126 JupyterHub log:174] 201 POST /hub/api/users/testuser-7c70eb90-035b-4d9f-92a5-482e441e307d (test@127.0.0.1) 20.74ms
+        [I 2019-09-25 12:03:12.153 JupyterHub log:174] 200 GET /hub/api/users/testuser-7c70eb90-035b-4d9f-92a5-482e441e307d (test@127.0.0.1) 11.91ms
+        [D 2019-09-25 12:03:12.180 JupyterHub user:240] Creating <class 'kubespawner.spawner.KubeSpawner'> for testuser-7c70eb90-035b-4d9f-92a5-482e441e307d:
+        [I 2019-09-25 12:03:12.204 JupyterHub reflector:199] watching for pods with label selector='component=singleuser-server' in namespace jh-dev
+        [D 2019-09-25 12:03:12.205 JupyterHub reflector:202] Connecting pods watcher
+        [I 2019-09-25 12:03:12.229 JupyterHub reflector:199] watching for events with field selector='involvedObject.kind=Pod' in namespace jh-dev
+        [D 2019-09-25 12:03:12.229 JupyterHub reflector:202] Connecting events watcher
+        [I 2019-09-25 12:03:12.269 JupyterHub log:174] 204 DELETE /hub/api/users/testuser-7c70eb90-035b-4d9f-92a5-482e441e307d (test@127.0.0.1) 98.85ms
+    """
+    print("create a user, and get information about the user")
+    r = api_request.get("/users/" + jupyter_user)
+    assert r.status_code == 200
+    assert r.json()["name"] == jupyter_user
+
+
+def test_api_list_users(api_request, jupyter_user):
+    """
+    Tests the hub api's /users endpoint. Information about users should be
+    returned.
+
+    A typical jupyterhub logging response to this test:
+
+        [I 2019-09-25 12:03:12.303 JupyterHub log:174] 201 POST /hub/api/users/testuser-0d2b0fc9-5ac4-4d8c-8d25-c4545665f81f (test@127.0.0.1) 15.53ms
+        [I 2019-09-25 12:03:12.331 JupyterHub log:174] 200 GET /hub/api/users (test@127.0.0.1) 10.83ms
+        [D 2019-09-25 12:03:12.358 JupyterHub user:240] Creating <class 'kubespawner.spawner.KubeSpawner'> for testuser-0d2b0fc9-5ac4-4d8c-8d25-c4545665f81f:
+        [I 2019-09-25 12:03:12.365 JupyterHub log:174] 204 DELETE /hub/api/users/testuser-0d2b0fc9-5ac4-4d8c-8d25-c4545665f81f (test@127.0.0.1) 18.44ms
+    """
+
+    print("create a test user, get information about all users, and find the test user")
+    r = api_request.get("/users")
+    assert r.status_code == 200
+    assert any(u["name"] == jupyter_user for u in r.json())
+
+
+def test_api_proxy_connectivity(api_request, request_data):
+    """
+    Tests the hub api's /proxy endpoint.
+
+    A typical jupyterhub logging response to this test:
+
+        [I 2019-09-25 12:03:12.395 JupyterHub log:174] 200 GET /hub/api/proxy (test@127.0.0.1) 13.48ms
+    """
+
+    endtime = time.time() + request_data["test_timeout"]
+    while time.time() < endtime:
+        try:
+            r = api_request.get("/proxy")
+            if r.status_code == 200:
+                break
+            print(r.json())
+        except requests.RequestException as e:
+            print(e)
+        time.sleep(1)
+    assert r.status_code == 200, "Failed to get /proxy"
+
+
+def test_extra_files(extra_files_test_command):
+    """
+    Tests the hub.extraFiles configuration. It should have mounted files to the
+    hub pod's container with specific file system permissions.
+    """
+    c = subprocess.run(
+        [
+            "kubectl",
+            "exec",
+            "deploy/hub",
+            "--",
+            "sh",
+            "-c",
+            extra_files_test_command,
+        ]
+    )
+    assert (
+        c.returncode == 0
+    ), f"The hub.extraFiles configuration doesn't seem to have been honored!"
+
+
+def test_load_etc_jupyterhub_d():
+    """
+    Tests that the extra jupyterhub config file put into
+    /usr/local/etc/jupyterhub/jupyterhub_config.d by the hub.extraFiles
+    configuration was loaded.
+    """
+    c = subprocess.run(
+        [
+            "kubectl",
+            "exec",
+            "deploy/hub",
+            "--",
+            "sh",
+            "-c",
+            "cat /tmp/created-by-extra-files-config.txt | grep -- 'hello world' || exit 1",
+        ]
+    )
+    assert (
+        c.returncode == 0
+    ), f"The hub.extraFiles configuration should have mounted a config file to /usr/local/etc/jupyterhub/jupyterhub_config.d which should have been loaded to write a dummy file for us!"
+
+
+def test_load_existing_secret():
+    """
+    Tests that use of hub.existingSecret works as intended, which mean the
+    JupyterHub helm chart should make use of the combined values from a self
+    managed k8s Secret and a chart managed k8s Secret. The one exception to this
+    rule is CONFIGPROXY_AUTH_TOKEN which is always set using the chart managed
+    k8s Secret.
+
+    Required for this test to not error or be skipped:
+
+        1. The chart is configured with a hub.extraConfig entry emitting logs
+           that this test looks for in the hub logs. Test error otherwise.
+        2. The chart is configured with hub.existingSecret. Skip otherwise.
+
+    For this test to work, we rely on:
+
+        - .github/workflows/test-chart.yaml to pass matrix.local-chart-extra-args
+          to the helm install command, and for it to set
+          matrix.create-k8s-test-resources on the same GitHub workflow job.
+        - ci/test-hub-existing-secret.yaml to define a k8s Secret to create.
+        - dev-config.yaml to set hub.services.test-hub-existing-secret.apiToken,
+          and for it to set hub.extraConfig to log values on jupyterhub startup.
+    """
+    c = subprocess.run(
+        [
+            "kubectl",
+            "logs",
+            "deploy/hub",
+        ],
+        text=True,
+        stdout=subprocess.PIPE,
+        stderr=subprocess.STDOUT,
+    )
+    hub_logs = c.stdout
+
+    if c.returncode != 0:
+        print(f"Return code: {c.returncode}")
+        print("---")
+        print(hub_logs)
+        raise AssertionError("Logs from deploy/hub not accessible!")
+
+    # error on missing hub.extraConfig entry
+    assert (
+        "test_hub_existing_secret" in hub_logs
+    ), "Expected specific logs to be emitted as planned from dev-config.yaml!"
+
+    # skip on missing hub.existingSecret or the referenced secret isn't available
+    if "hub.existingSecret=None" in hub_logs:
+        pytest.skip("hub.existingSecret is None")
+    else:
+        k8s_secret_exist = False
+        match = re.compile(r".*hub.existingSecret=(?P<ref>[\S]*).*", re.DOTALL).match(
+            hub_logs
+        )
+        if match:
+            k8s_secret = match.group("ref")
+            assert k8s_secret, "This should never be found "
+            c = subprocess.run(["kubectl", "get", "secret", k8s_secret])
+            if c.returncode != 0:
+                pytest.skip(f"k8s Secret '{k8s_secret}' not found")
+
+    # NOTE:  ffff9999 are values from ci/test-hub-existing-secret.yaml that
+    #        hub.existingSecret will reference, while aaaa1111-dddd4444 come from
+    #        the chart managed k8s Secret.
+    #
+    # FIXME: It would be good to test use against custom databases, then we
+    #        would also test hub.db.password here and that the environment
+    #        variables MYSQL_PWD and PGPASSWORD would be setup correctly. But,
+    #        configuring that would make the hub fail to startup without an
+    #        actual database.
+    assert "hub.services.test-hub-existing-secret.apiToken=ffff9999" in hub_logs
+    assert "CONFIGPROXY_AUTH_TOKEN=aaaa1111" in hub_logs
+    assert "JupyterHub.cookie_secret=ffff9999" in hub_logs
+    assert "CryptKeeper.keys=ffff9999" in hub_logs
+    assert "singleuser.extraLabels.test-chart-managed-secret=ok" in hub_logs
+    assert "singleuser.extraLabels.test-self-managed-secret=ok" in hub_logs
diff --git a/tests/test_spawn.py b/tests/test_spawn.py
--- a/tests/test_spawn.py
+++ b/tests/test_spawn.py
@@ -1,186 +1,21 @@
-import os
 import subprocess
 import time
 
 import pytest
 import requests
-import yaml
 
-# Makes heavy use of JupyterHub's API:
-# http://petstore.swagger.io/?url=https://raw.githubusercontent.com/jupyterhub/jupyterhub/master/docs/rest-api.yml
 
-# load app version of chart
-here = os.path.dirname(os.path.abspath(__file__))
-chart_yaml = os.path.join(here, os.pardir, "jupyterhub", "Chart.yaml")
-
-extra_files_test = """
-    ls -l /tmp/binaryData.txt | grep -- -rw-rw-rw- || exit 1
-    ls -l /tmp/dir1/binaryData.txt | grep -- -rw-rw-rw- || exit 2
-    ls -l /tmp/stringData.txt | grep -- -rw-rw-rw- || exit 3
-    ls -l /tmp/dir1/stringData.txt | grep -- -rw-rw-rw- || exit 4
-    ls -l /etc/test/data.yaml | grep -- -r--r--r-- || exit 5
-    ls -l /etc/test/data.yml | grep -- -r--r--r-- || exit 6
-    ls -l /etc/test/data.json | grep -- -r--r--r-- || exit 7
-    ls -l /etc/test/data.toml | grep -- -r--r--r-- || exit 8
-    cat /tmp/binaryData.txt | grep -- "hello world" || exit 9
-    cat /tmp/stringData.txt | grep -- "hello world" || exit 10
-"""
-
-with open(chart_yaml) as f:
-    chart = yaml.safe_load(f)
-    jupyterhub_version = chart["appVersion"]
-
-
-def test_api(api_request):
-    """
-    Tests the hub api's root endpoint (/). The hub's version should be returned.
-
-    A typical jupyterhub logging response to this test:
-
-        [I 2019-09-25 12:03:12.051 JupyterHub log:174] 200 GET /hub/api (test@127.0.0.1) 9.57ms
-    """
-
-    print("asking for the hub's version")
-    r = api_request.get("")
-    assert r.status_code == 200
-    assert r.json().get("version", "version-missing") == jupyterhub_version
-
-
-def test_api_info(api_request):
-    """
-    Tests the hub api's /info endpoint. Information about the hub should be
-    returned.
-
-    A typical jupyterhub logging response to this test:
-
-        [I 2019-09-25 12:03:12.086 JupyterHub log:174] 200 GET /hub/api/info (test@127.0.0.1) 10.21ms
-    """
-
-    print("asking for the hub information")
-    r = api_request.get("/info")
-    assert r.status_code == 200
-    result = r.json()
-    assert result["spawner"]["class"] == "kubespawner.spawner.KubeSpawner"
-
-
-def test_hub_api_create_user_and_get_information_about_user(api_request, jupyter_user):
-    """
-    Tests the hub api's /users/:user endpoint, both POST and GET.
-
-    A jupyter user is created and commited to the hub database through the
-    jupyter_user pytest fixture declared in conftest.py. Due to this, this first
-    test to use the jupyter_user fixture is actually testing both the pytest
-    fixture that creates the user as well as the ability to get information from
-    the hub about the user. The jupyter_user fixture will automatically clean up
-    the user from the hub's database when this function exits.
-
-    A typical jupyterhub logging response to this test:
-
-        [I 2019-09-25 12:03:12.126 JupyterHub log:174] 201 POST /hub/api/users/testuser-7c70eb90-035b-4d9f-92a5-482e441e307d (test@127.0.0.1) 20.74ms
-        [I 2019-09-25 12:03:12.153 JupyterHub log:174] 200 GET /hub/api/users/testuser-7c70eb90-035b-4d9f-92a5-482e441e307d (test@127.0.0.1) 11.91ms
-        [D 2019-09-25 12:03:12.180 JupyterHub user:240] Creating <class 'kubespawner.spawner.KubeSpawner'> for testuser-7c70eb90-035b-4d9f-92a5-482e441e307d:
-        [I 2019-09-25 12:03:12.204 JupyterHub reflector:199] watching for pods with label selector='component=singleuser-server' in namespace jh-dev
-        [D 2019-09-25 12:03:12.205 JupyterHub reflector:202] Connecting pods watcher
-        [I 2019-09-25 12:03:12.229 JupyterHub reflector:199] watching for events with field selector='involvedObject.kind=Pod' in namespace jh-dev
-        [D 2019-09-25 12:03:12.229 JupyterHub reflector:202] Connecting events watcher
-        [I 2019-09-25 12:03:12.269 JupyterHub log:174] 204 DELETE /hub/api/users/testuser-7c70eb90-035b-4d9f-92a5-482e441e307d (test@127.0.0.1) 98.85ms
-    """
-    print("create a user, and get information about the user")
-    r = api_request.get("/users/" + jupyter_user)
-    assert r.status_code == 200
-    assert r.json()["name"] == jupyter_user
-
-
-def test_hub_api_list_users(api_request, jupyter_user):
-    """
-    Tests the hub api's /users endpoint. Information about users should be
-    returned.
-
-    A typical jupyterhub logging response to this test:
-
-        [I 2019-09-25 12:03:12.303 JupyterHub log:174] 201 POST /hub/api/users/testuser-0d2b0fc9-5ac4-4d8c-8d25-c4545665f81f (test@127.0.0.1) 15.53ms
-        [I 2019-09-25 12:03:12.331 JupyterHub log:174] 200 GET /hub/api/users (test@127.0.0.1) 10.83ms
-        [D 2019-09-25 12:03:12.358 JupyterHub user:240] Creating <class 'kubespawner.spawner.KubeSpawner'> for testuser-0d2b0fc9-5ac4-4d8c-8d25-c4545665f81f:
-        [I 2019-09-25 12:03:12.365 JupyterHub log:174] 204 DELETE /hub/api/users/testuser-0d2b0fc9-5ac4-4d8c-8d25-c4545665f81f (test@127.0.0.1) 18.44ms
-    """
-
-    print("create a test user, get information about all users, and find the test user")
-    r = api_request.get("/users")
-    assert r.status_code == 200
-    assert any(u["name"] == jupyter_user for u in r.json())
-
-
-def test_hub_can_talk_to_proxy(api_request, request_data):
-    """
-    Tests the hub api's /proxy endpoint.
-
-    A typical jupyterhub logging response to this test:
-
-        [I 2019-09-25 12:03:12.395 JupyterHub log:174] 200 GET /hub/api/proxy (test@127.0.0.1) 13.48ms
-    """
-
-    endtime = time.time() + request_data["test_timeout"]
-    while time.time() < endtime:
-        try:
-            r = api_request.get("/proxy")
-            if r.status_code == 200:
-                break
-            print(r.json())
-        except requests.RequestException as e:
-            print(e)
-        time.sleep(1)
-    assert r.status_code == 200, "Failed to get /proxy"
-
-
-def test_hub_mounted_extra_files():
-    """
-    Tests the hub.extraFiles configuration. It should have mounted files to the
-    hub pod's container with specific file system permissions.
-    """
-    c = subprocess.run(
-        [
-            "kubectl",
-            "exec",
-            "deploy/hub",
-            "--",
-            "sh",
-            "-c",
-            extra_files_test,
-        ]
-    )
-    assert (
-        c.returncode == 0
-    ), f"The hub.extraFiles configuration doesn't seem to have been honored!"
-
-
-def test_hub_etc_jupyterhub_d_folder():
-    """
-    Tests that the extra jupyterhub config file put into
-    /usr/local/etc/jupyterhub/jupyterhub_config.d by the hub.extraFiles
-    configuration was loaded.
-    """
-    c = subprocess.run(
-        [
-            "kubectl",
-            "exec",
-            "deploy/hub",
-            "--",
-            "sh",
-            "-c",
-            "cat /tmp/created-by-extra-files-config.txt | grep -- 'hello world' || exit 1",
-        ]
-    )
-    assert (
-        c.returncode == 0
-    ), f"The hub.extraFiles configuration should have mounted a config file to /usr/local/etc/jupyterhub/jupyterhub_config.d which should have been loaded to write a dummy file for us!"
-
-
-def test_hub_api_request_user_spawn(
-    api_request, jupyter_user, request_data, pebble_acme_ca_cert
+def test_spawn_basic(
+    api_request,
+    jupyter_user,
+    request_data,
+    pebble_acme_ca_cert,
+    extra_files_test_command,
 ):
     """
     Tests the hub api's /users/:user/server POST endpoint. A user pod should be
-    created with environment variables defined in singleuser.extraEnv etc.
+    created with environment variables defined in singleuser.extraEnv,
+    singleuser.extraFiles should be mounted, etc.
     """
 
     print("asking kubespawner to spawn a server for a test user")
@@ -227,7 +62,7 @@ def test_hub_api_request_user_spawn(
                 "--",
                 "sh",
                 "-c",
-                extra_files_test,
+                extra_files_test_command,
             ]
         )
         assert (
@@ -238,7 +73,7 @@ def test_hub_api_request_user_spawn(
 
 
 @pytest.mark.netpol
-def test_singleuser_netpol(api_request, jupyter_user, request_data):
+def test_spawn_netpol(api_request, jupyter_user, request_data):
     """
     Tests a spawned user pods ability to communicate with allowed and blocked
     internet locations.