Repository: jupyterhub/zero-to-jupyterhub-k8s

Issue:
Enable injection of files to the singleuser pods and the hub pod
## Motivation
This is meant to support a quite wide range of applications where mounting a file is relevant.

- https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/1987 and https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/1580 suggests to mount files while also making hub.extraConfig load them. I suggest this is done addressed by mounting to a specific folder.
- The ability to mount a `jupyter_notebook_config.json` specifically to /etc/jupyter for user pods, see: https://github.com/jupyterhub/zero-to-jupyterhub-k8s/pull/1691
- The ability to mount templates for jupyterhub, see: https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/1266

## Chart configuration idea
We expose configuration both under hub and singleuser, for example under `extraFiles`. We make it an dictionary just like `hub.extraConfig`, and let that dictionary's keys be arbitrary names and let the values be dictionaries with the file information.

```yaml
hub:
  extraFiles:
    kitten.png:
      mountPath: /usr/local/share/jupyterhub/static/kitten.png
      # A base64 encoding representing the binary content indirectly
      binaryData: |
        c29tZSBiaW5hcnkgY29udGVudCBiYXNlNjQgZW5jb2RlZAo=

singleuser:
  extraFiles:
    jupyter_notebook_config.json:
      mountPath: /etc/jupyter/jupyter_notebook_config.json
      # After this value is parsed from YAML (including JSON), we must render it back to a string.
      # At this point, we can opt to render toJson, toYaml, or toToml based on the file name extension.
      data:
        NotebookApp:
          allow_origin: '*'
          tornado_settings:
            trust_xheaders: true
          shutdown_no_activity_timeout: 600
    gitconfig:
      mountPath: /etc/gitconfig
      mode: 0644
      # A string representing the file content directly
      stringData: |
        [alias]
        # git l - an alias for a condensed view of commits
        # CREDITS: https://ma.ttias.be/pretty-git-log-in-one-line/
        l = log --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' --abbrev-commit
```

## Implementation
The general idea is to add a volume and a volumeMount to the pod in question, but there are some deliberations along the way.

- __Use subPath?__
  Using subPath allow you to mount a specific key from a k8s configmap/secret into a specific file, rather than mounting all keys of a configmap/secret into a directory. By using subPath, the mount will no longer update on changes to the configmap/secret, but by not using subPath you must will erase all other content in the mount directory.
  - [x] Decision
    Rely entirely on usage of subPath, supporting dynamic updates for files injected into a folder is just a bit too much overhead and requires initContainers etc.
- __Use ConfigMaps or Secrets?__
  - ConfigMaps are easier to inspect by being written in plain text, while Secrets are base64 encoded.
  - Secret's base64 encoding will make content take 4/3 of the space.
  - Secret's base64 encoding will allow for binary file mounts such as images for hub templates.
  - Secret's is a safer bet from a permission standpoint.
  - [x] Decision
    Rely entirely on one or more k8s Secrets, don't complicate it by sometimes using a k8s ConfigMap.
- __How many Secrets should be used to pass the files?__
  - One single, one per file, or one default + a dynamic amount of additional ones? The benefit of having multiple is that more data can be fit into them. Using a single file, we are limited to about ~3 MB of data due to k8s / etcd constraints.
  - [x] Decision
    At this point, assume a single which won't shut the door from allowing a dynamic amount of additional ones later.
- __What k8s Secret resources for hub/singleuser should be used?__
  - Option A: a new dedicated k8s Secret containing both hub/singleuser extraFiles?
  - Option B: two new dedicated k8s Secrets containing hub and singleuser extraFiles repsectively?
  - Option C: re-use the hub k8s Secret to contain both hub/singleuser extraFiles?
  - Option D: re-use the hub k8s Secret and create a new dedicated k8s Secret for singleuser.
  - [x] Decision
    I'm going towards Option D for now.
- __Configuration option name?__
  - Is extraFiles good? I like that it includes files in the name, but perhaps being more explicit and saying extraFileMounts is worth considering, hinting it will influence volumeMounts (and volumes) of a pod spec also.
  - [x] Decision
    hub.extraFiles / singleuser.extraFiles it is!
  - Is mountPath / data / stringData / binaryData good?
  - [x] Decision
    I think so, let's go with it for now.
- __Implicit and/or explicit data format?__
  - When the user passes `data` as a data structure, we will in the end need to render it to a string. Helm provides us with `toYaml`, `toToml`, and `toJson` to do that. We could decide what data structure to render to using the filename extension, but perhaps there is an edge case where we need to support an explicit data format? If we need an explicit format, we could name it `dataFormat` where `yaml`, `json`, `toml` would be the valid options.
  - [x] Decision
    For now let's only support the implicit format and let Helm template rendering fail if a filename is passed without a supported filename extension when `data` is used.
- __Expose mode?__
  When declaring a SecretVolumeSource as a Volume you can set `defaultMode` such as 0644, and you can set an explicit map of files and `mode` for them. It can be critical for some files, such as SSH keys.
  - [x] Decision
    I'll try implement `mode` on a file to file basis and excluding `defaultMode` which I think will be fine as it is, and would cause complications with the decision to re-use the hub k8s Secret.
- __Explicit name notation?__
  As extraFiles declare files in many different folders, the having the name as a key could lead to name conflicts. Due to that, it would make sense to have a name field next to `mountPath` allowing multiple files of the same name to be mounted to different folders.
  - [x] Decision
    I'll implement it! For reference it's worth considering [hub/singleuser.extraEnv](https://zero-to-jupyterhub.readthedocs.io/en/latest/resources/reference.html#hub-extraenv) also provides this.
    
### On passing files
Its straightforward on how to pass dictionary like configuration, but, how to go about passing a large `.py` file or a binary image file?

__A standalone text file__
Just pass `--set-file singleuser.extraFiles.my_file_key.stringData=./my_file.py` to the helm upgrade command.

__A standalone binary file__
With binary files like an image, using `--set-file` wouldn't work as it doesn't know how to convert it to a string, which it must be. This is a challenge.

There is a PR open for `--include-file` etc which would make files accessible to the Helm template rendering through `.Files` which would save the day I think. It is being considered for Helm v3.6 at the earliest.

It would be possible for users to base64 encode their files and then use --set-file on the base64 encoded version and then passing `--set-file singleuser.extraFiles.my_image.binaryData=./my_image_base64_encoded.txt`.

__Decision__
For now, we don't create some suggested solution to mounting standalone binary files directly but instead await if for example https://github.com/helm/helm/pull/8841 becomes included in a future version of Helm.

## References
- Kubernetes API reference: [Secret](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#secret-v1-core), [ConfigMap](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#configmap-v1-core), [SecretVolumeSource](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#secretvolumesource-v1-core), [Volume](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#volume-v1-core), [VolumeMount](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#volumemount-v1-core).


Discussion:
I think this will be a truly awesome improvement for dependent charts like binderhub/mybinder.org. I've used this in some of my deployments with great effect, and if we can get support in the base chart, that's awesome.

> Is extraFiles good?

I like extraFiles. I don't think there's any ambiguity that the files will be mounted.

> Is mountPath / data / stringData / binaryData good?

What do the precedents in secrets/configmaps look like? How are these distinguished now?


> > Is mountPath / data / stringData / binaryData good?
> 
> What do the precedents in secrets/configmaps look like? How are these distinguished now?

- [ConfigMaps](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#configmap-v1-core) have `data` and `binaryData` fields.
- [Secrets](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#secret-v1-core) have `data` and `stringData` fields.

`data` in ConfigMaps equals `stringData` in Secrets and `data` in Secrets equals `binaryData` in ConfigMaps. They are always strings, but `binaryData` is assumed to be a base64 encoded string.

If we ignore k8s notations, nice things to convey are:
1. `data` should be a dictionary when interpreted by Helm. We will render it with toYaml/toJson/toToml based on something, like the filename extension.
2. `stringData` should be a string representing the content of a string file
3. `binaryData` should be a base64 encoding of a binary blob of the actual binary file

Resolving patch:
diff --git a/jupyterhub/files/hub/jupyterhub_config.py b/jupyterhub/files/hub/jupyterhub_config.py
--- a/jupyterhub/files/hub/jupyterhub_config.py
+++ b/jupyterhub/files/hub/jupyterhub_config.py
@@ -235,7 +235,6 @@ def camelCaseify(s):
 
 # Configure dynamically provisioning pvc
 storage_type = get_config("singleuser.storage.type")
-
 if storage_type == "dynamic":
     pvc_name_template = get_config("singleuser.storage.dynamic.pvcNameTemplate")
     c.KubeSpawner.pvc_name_template = pvc_name_template
@@ -280,6 +279,43 @@ def camelCaseify(s):
         }
     ]
 
+# Inject singleuser.extraFiles as volumes and volumeMounts with data loaded from
+# the dedicated k8s Secret prepared to hold the extraFiles actual content.
+extra_files = get_config("singleuser.extraFiles", {})
+if extra_files:
+    volume = {
+        "name": "files",
+    }
+    items = []
+    for file_key, file_details in extra_files.items():
+        # Each item is a mapping of a key in the k8s Secret to a path in this
+        # abstract volume, the goal is to enable us to set the mode /
+        # permissions only though so we don't change the mapping.
+        item = {
+            "key": file_key,
+            "path": file_key,
+        }
+        if "mode" in file_details:
+            item["mode"] = file_details["mode"]
+        items.append(item)
+    volume["secret"] = {
+        "secretName": get_name("singleuser"),
+        "items": items,
+    }
+    c.KubeSpawner.volumes.append(volume)
+
+    volume_mounts = []
+    for file_key, file_details in extra_files.items():
+        volume_mounts.append(
+            {
+                "mountPath": file_details["mountPath"],
+                "subPath": file_key,
+                "name": "files",
+            }
+        )
+    c.KubeSpawner.volume_mounts.extend(volume_mounts)
+
+# Inject extraVolumes / extraVolumeMounts
 c.KubeSpawner.volumes.extend(get_config("singleuser.storage.extraVolumes", []))
 c.KubeSpawner.volume_mounts.extend(
     get_config("singleuser.storage.extraVolumeMounts", [])
@@ -371,6 +407,15 @@ def camelCaseify(s):
     c.JupyterHub.log_level = "DEBUG"
     c.Spawner.debug = True
 
+# load /usr/local/etc/jupyterhub/jupyterhub_config.d config files
+config_dir = "/usr/local/etc/jupyterhub/jupyterhub_config.d"
+if os.path.isdir(config_dir):
+    for file_name in sorted(os.listdir(config_dir)):
+        print(f"Loading {config_dir} config: {file_name}")
+        with open(f"{config_dir}/{file_name}") as f:
+            file_content = f.read()
+        # compiling makes debugging easier: https://stackoverflow.com/a/437857
+        exec(compile(source=file_content, filename=file_name, mode="exec"))
 
 # load potentially seeded secrets
 c.JupyterHub.proxy_auth_token = get_secret_value("JupyterHub.proxy_auth_token")
diff --git a/jupyterhub/files/hub/z2jh.py b/jupyterhub/files/hub/z2jh.py
--- a/jupyterhub/files/hub/z2jh.py
+++ b/jupyterhub/files/hub/z2jh.py
@@ -15,7 +15,7 @@ def _load_config():
     """Load the Helm chart configuration used to render the Helm templates of
     the chart from a mounted k8s Secret."""
 
-    path = f"/etc/jupyterhub/secret/values.yaml"
+    path = f"/usr/local/etc/jupyterhub/secret/values.yaml"
     if os.path.exists(path):
         print(f"Loading {path}")
         with open(path) as f:
@@ -28,7 +28,7 @@ def _load_config():
 def _get_config_value(key):
     """Load value from the k8s ConfigMap given a key."""
 
-    path = f"/etc/jupyterhub/config/{key}"
+    path = f"/usr/local/etc/jupyterhub/config/{key}"
     if os.path.exists(path):
         with open(path) as f:
             return f.read()
@@ -40,7 +40,7 @@ def _get_config_value(key):
 def get_secret_value(key):
     """Load value from the k8s Secret given a key."""
 
-    path = f"/etc/jupyterhub/secret/{key}"
+    path = f"/usr/local/etc/jupyterhub/secret/{key}"
     if os.path.exists(path):
         with open(path) as f:
             return f.read()


Test patch:
diff --git a/tests/test_spawn.py b/tests/test_spawn.py
--- a/tests/test_spawn.py
+++ b/tests/test_spawn.py
@@ -13,6 +13,19 @@
 here = os.path.dirname(os.path.abspath(__file__))
 chart_yaml = os.path.join(here, os.pardir, "jupyterhub", "Chart.yaml")
 
+extra_files_test = """
+    ls -l /tmp/binaryData.txt | grep -- -rw-rw-rw- || exit 1
+    ls -l /tmp/dir1/binaryData.txt | grep -- -rw-rw-rw- || exit 2
+    ls -l /tmp/stringData.txt | grep -- -rw-rw-rw- || exit 3
+    ls -l /tmp/dir1/stringData.txt | grep -- -rw-rw-rw- || exit 4
+    ls -l /etc/test/data.yaml | grep -- -r--r--r-- || exit 5
+    ls -l /etc/test/data.yml | grep -- -r--r--r-- || exit 6
+    ls -l /etc/test/data.json | grep -- -r--r--r-- || exit 7
+    ls -l /etc/test/data.toml | grep -- -r--r--r-- || exit 8
+    cat /tmp/binaryData.txt | grep -- "hello world" || exit 9
+    cat /tmp/stringData.txt | grep -- "hello world" || exit 10
+"""
+
 with open(chart_yaml) as f:
     chart = yaml.safe_load(f)
     jupyterhub_version = chart["appVersion"]
@@ -119,6 +132,49 @@ def test_hub_can_talk_to_proxy(api_request, request_data):
     assert r.status_code == 200, "Failed to get /proxy"
 
 
+def test_hub_mounted_extra_files():
+    """
+    Tests the hub.extraFiles configuration. It should have mounted files to the
+    hub pod's container with specific file system permissions.
+    """
+    c = subprocess.run(
+        [
+            "kubectl",
+            "exec",
+            "deploy/hub",
+            "--",
+            "sh",
+            "-c",
+            extra_files_test,
+        ]
+    )
+    assert (
+        c.returncode == 0
+    ), f"The hub.extraFiles configuration doesn't seem to have been honored!"
+
+
+def test_hub_etc_jupyterhub_d_folder():
+    """
+    Tests that the extra jupyterhub config file put into
+    /usr/local/etc/jupyterhub/jupyterhub_config.d by the hub.extraFiles
+    configuration was loaded.
+    """
+    c = subprocess.run(
+        [
+            "kubectl",
+            "exec",
+            "deploy/hub",
+            "--",
+            "sh",
+            "-c",
+            "cat /tmp/created-by-extra-files-config.txt | grep -- 'hello world' || exit 1",
+        ]
+    )
+    assert (
+        c.returncode == 0
+    ), f"The hub.extraFiles configuration should have mounted a config file to /usr/local/etc/jupyterhub/jupyterhub_config.d which should have been loaded to write a dummy file for us!"
+
+
 def test_hub_api_request_user_spawn(
     api_request, jupyter_user, request_data, pebble_acme_ca_cert
 ):
@@ -161,6 +217,22 @@ def test_hub_api_request_user_spawn(
         assert (
             c.returncode == 0
         ), f"singleuser.extraEnv didn't lead to a mounted environment variable!"
+
+        # check user pod's extra files
+        c = subprocess.run(
+            [
+                "kubectl",
+                "exec",
+                pod_name,
+                "--",
+                "sh",
+                "-c",
+                extra_files_test,
+            ]
+        )
+        assert (
+            c.returncode == 0
+        ), f"The singleuser.extraFiles configuration doesn't seem to have been honored!"
     finally:
         _delete_server(api_request, jupyter_user, request_data["test_timeout"])