Repository: jupyterhub/zero-to-jupyterhub-k8s

Issue:
Set singleuser.args to jupyterhub-singleuser
After writing
https://github.com/pangeo-data/pangeo-docker-images/pull/355/files#diff-a77643b43a7be453fa8556937bf32b27907e152a10d4c693f3e7670c66a44378,
I would like to ask that we reconsider *requiring* entrypoint be set to
a specific thing (`jupyterhub-singleuser`) in the images users are
required to produce. This makes it quite difficult to produce images
that *work across multiple tools* without having to configure each of
those tools. In this case, we want to use the *exact same image* in both
binder and apache beam. Apache beam *requires* an entrypoint be
specified, while at least as of now binder does not - so I wrote a
script that uses heuristics to determine if the caller is apache beam or
something else, and change behavior accordingly. But if I want this
image to also work with JupyterHub in the future, I'll have to modify my
script to detect if JupyterHub is calling it, and exec
`jupyterhub-singleuser`. Now if dask also wants to use this image, I'd
need to have a code path for that as well. I think this strong coupling
causes a lot of unnecessary friction here.

Another point of confusion here is that `cmd` in helm / k8s is actually
*ENTRYPOINT* in docker, while `args` in helm / k8s is *CMD* in docker.
Earlier, we were setting `cmd` to `jupyterhub-singleuser`, which
overrode any entrypoint the docker builder had set up - causing the
issues we were trying to fix. However, now that KubeSpawner doesn't
actually use `args` (k8s / helm) in any meaningful fashion, can we set
the default `args` to `jupyterhub-singleuser`? This has a few
advantages:

1. *ENTRYPOINT* set in Dockerfile is *not* overriden by what we set in
   helm, which is the current problem. This goes away
2. All existing docker images continue to work without admins having to
   set extra config or the image having to be rebuilt.
3. The same image can be used in multiple places without having to be
   aware of who is calling it.

I think this is an approach to solving the original problem that is both
*easier to document*, *easier for our users (existing and new)* and
makes life a lot less difficult for people building images.

Re-implementation of https://github.com/jupyterhub/zero-to-jupyterhub-k8s/pull/2449


Discussion:
I can investigate the test failures and try getting this to work if y'all think the general approach is workable, @manics @minrk @consideRatio
Here's the somewhat tortured test I had to write to validate my 'multi dispatch entrypoint' works ok https://github.com/pangeo-data/pangeo-docker-images/pull/355/files#diff-6027b5fe95c80f9fe0bcead4ae1288105dfee85005ad12d2aa07d58c0d2fa8d2R35
/cc @minrk @consideRatio @manics 
The aim of the original PR was to ensure images that run scripts with custom setup will work without changes. For example, the docker-stacks images run some hooks to customise the environment before running jupyterhub-singleuser:
https://discourse.jupyter.org/t/pyspark-library-is-missing-from-jupyter-pyspark-notebook-when-running-with-jupyterhub-zero-to-jupyterhub-k8s/8450/2

If we override the entrypoint or command this will break the docker-stacks images again:
https://github.com/jupyter/docker-stacks/blob/57ebe55c07d06df70f6fa1166bc5d8ce22b6da34/base-notebook/Dockerfile#L144


### About docker-stacks images

Docker stacks relies on the startup scripts to do misc logic, but will call `jupyterhub-singleuser` if a `JUPYTERHUB_API_TOKEN` env var is found. This is done in three steps: [step 1 - start-notebook.sh](https://github.com/jupyter/docker-stacks/blob/57ebe55c07d06df70f6fa1166bc5d8ce22b6da34/base-notebook/start-notebook.sh#L11-L14), [step 2 - start-singleuser.sh](https://github.com/jupyter/docker-stacks/blob/57ebe55c07d06df70f6fa1166bc5d8ce22b6da34/base-notebook/start-singleuser.sh#L12-L13), [step 3 - start.sh](https://github.com/jupyter/docker-stacks/blob/57ebe55c07d06df70f6fa1166bc5d8ce22b6da34/base-notebook/start.sh#L260-L261).

From a docker-stacks perspective, such images should be started with the start-notebook.sh script if users also want to rely on various environment variable influencing those scripts. They can start like that either by default via their Dockerfile's declared ENTRYPOINT+CMD, or via z2jh's configuration of kubespawner, which in turn configures the k8s `args` (but not the k8s `command` aka. Dockerfile's ENTRYPOINT).

### About reverting #2449

There is a key part that I know @minrk wants resolved, and that is the possibility to configuring z2jh in a way that makes the Dockerfile's CMD be respected. Can we do that? I recall an issue, so I'll look into this...

Currently, `singleuser.cmd` configuration in z2jh is used like this:

https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/58d7d67192930b95bc1ea14542aabbf04d2605b3/jupyterhub/files/hub/jupyterhub_config.py#L412-L413

[The default value of `cmd` in KubeSpawner](https://github.com/jupyterhub/kubespawner/blob/9f8a0f5364bc64d1302c6ea8b6602bca3505b422/kubespawner/spawner.py#L430-L445) is None. Good!

Okay, I think we should be good. Users that want the behavior of relying on the `CMD` in the Dockerfiles can specify `singleuser.cmd=null` and by doing so override the z2jh default configuration of `jupyterhub-singleuser`. If a user wants certain profile options to rely on the image, they can also do that by setting the KubeSpawner instance's `cmd` traitlet to None as well.

### My current opinion

I think either choice, of defaulting to `jupyterhub-singleuser` as a `args`/`CMD`, or defaulting to what the image specifies is fine, and I'm not sure what I'd prefer most if this was setup from scratch as a new project.

Given that this isn't a new project, and we have a current default and have forced users that wanted to rely on `CMD` from their image to declare `singleuser.cmd=null`, I suggest that we stick with it and revert #2449. Why? Because its a breaking change that forces us to communicate and explain etc. For users that want to rely on `CMD` in image, it should be made clear that they can if they set `singleuser.cmd` or the KubeSpawner instance's `cmd` traitlet to `null` (YAML) or `None` (Python).

Example:

```yaml
singleuser:
  cmd:
  # declaring cmd like above is the same as declaring it like below
  # cmd: null
```
> Given that this isn't a new project, and we have a current default and have forced users that wanted to rely on `CMD` from their image to declare `singleuser.cmd=null`, I suggest that we stick with it and revert #2449. Why? Because its a breaking change that forces us to communicate and explain etc. For users that want to rely on `CMD` in image, it should be made clear that they can if they set `singleuser.cmd` or the KubeSpawner instance's `cmd` traitlet to `null` (YAML) or `None` (Python).

I'm very much +1 to this too. 
I've lost track of where it was discussed recently, but as long as set-to-null in a user chart to opt-in to the image's custom command is easy and doesn't require replicating the command itself (the original motivation of #2138, I think), I'm happy. As @consideRatio mentioned, changing None to be the default in KubeSpawner seeems to have made this possible, so with this change (after fixing `singleuser.args` to `singleuser.cmd`), we have:

- default behavior: always works, matches other JupyterHub contexts
- custom image CMD (i.e. docker stacks) is easy and clear to opt-in to: `singleuser.cmd = null` (solves #2138 without breaking other images or changing default behavior)

This seems the best balance to me.
OK, let's revert the breaking change then. Can you update https://zero-to-jupyterhub.readthedocs.io/en/latest/jupyterhub/customizing/user-environment.html#choose-and-use-an-existing-docker-image to include `singleuser.cmd: null` in the config?

Resolving patch:
diff --git a/jupyterhub/files/hub/jupyterhub_config.py b/jupyterhub/files/hub/jupyterhub_config.py
--- a/jupyterhub/files/hub/jupyterhub_config.py
+++ b/jupyterhub/files/hub/jupyterhub_config.py
@@ -408,8 +408,13 @@ def camelCaseify(s):
 
     c.JupyterHub.load_roles.append(role)
 
+# respect explicit null command (distinct from unspecified)
+# this avoids relying on KubeSpawner.cmd's default being None
+_unspecified = object()
+specified_cmd = get_config("singleuser.cmd", _unspecified)
+if specified_cmd is not _unspecified:
+    c.Spawner.cmd = specified_cmd
 
-set_config_if_not_none(c.Spawner, "cmd", "singleuser.cmd")
 set_config_if_not_none(c.Spawner, "default_url", "singleuser.defaultUrl")
 
 cloud_metadata = get_config("singleuser.cloudMetadata", {})


Test patch: