Repository: jupyter/docker-stacks

Issue:
[BUG] Healthcheck fails when using a custom runtime dir
### What docker image(s) are you using?

scipy-notebook (but applies to all images based on the `base-notebook` image)

### Host OS system

RHEL 8.0

### Host architecture

x86_64

### What Docker command are you running?

The following command DOES work as expected (default runtime dir):
```
docker run --rm -p 8888:8888 --name jupyter quay.io/jupyter/scipy-notebook:2023-12-25 start-notebook.sh
```

The following command does NOT work as expected (customized runtime dir):
```
docker run --rm -p 8888:8888 --name jupyter -e JUPYTER_RUNTIME_DIR=/home/jovyan/custom-runtime quay.io/jupyter/scipy-notebook:2023-12-25 start-notebook.sh
```

### How to Reproduce the problem?

1. Start the Jupyter container using the commands above.
2. In another terminal, run the healtcheck script: `docker exec jupyter /etc/jupyter/docker_healthcheck.py`
3. Observe the healthcheck script failing due to server state JSON file(s) not being found.

### Command output

```bash session
$ docker run --rm -p 8888:8888 --name jupyter quay.io/jupyter/scipy-notebook:2023-12-25 start-notebook.sh
$ docker exec jupyter /etc/jupyter/docker_healthcheck.py
b'{"version": "2.12.1"}'

$ docker run --rm -p 8888:8888 --name jupyter -e JUPYTER_RUNTIME_DIR=/home/jovyan/custom-runtime quay.io/jupyter/scipy-notebook:2023-12-25 start-notebook.sh
$ docker exec jupyter /etc/jupyter/docker_healthcheck.py
Traceback (most recent call last):
  File "/etc/jupyter/docker_healthcheck.py", line 14, in <module>
    json_file = next(runtime_dir.glob("*server-*.json"))
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
StopIteration
```


### Expected behavior

Healthcheck script to not fail, e.g. to display `b'{"version": "2.12.1"}'`, even with a customized runtime dir.

### Actual behavior

The healthcheck script fails because it cannot find server state JSON files in the hard-coded default runtime dir.

### Anything else?

The problem is that the `/etc/jupyter/docker_healthcheck.py` healtcheck script hard-codes the default runtime directory to search for server JSON state files as below:
https://github.com/jupyter/docker-stacks/blob/fcb20a914ed20e44a96053caf43eef6e12fb4c04/images/base-notebook/docker_healthcheck.py#L13

When this directory is customized for example via `JUPYTER_RUNTIME_DIR`, then the healthcheck script does not work.

The actual problem is when deploying Jupyter containers as services.
The Jupyter images have a default healthcheck configured as below:
https://github.com/jupyter/docker-stacks/blob/fcb20a914ed20e44a96053caf43eef6e12fb4c04/images/base-notebook/Dockerfile#L66-L70

When the healthcheck fails due to a custom runtime dir, the service is restarted continuously.

I think the healthcheck script should use the output of `jupyter --runtime-dir` which respects customizations:
```
$ docker run --rm -e JUPYTER_RUNTIME_DIR=/home/jovyan/custom-runtime quay.io/jupyter/scipy-notebook:2023-12-25 jupyter --runtime-dir
/home/jovyan/custom-runtime
```

If you agree with the above, I can send a PR with this fix.

### Latest Docker version

- [X] I've updated my Docker version to the latest available, and the issue persists


Discussion:
The workaround we are using for now is to disable healthchecks in the compose file. For example:
```yaml
services:
  notebook:
    image: quay.io/jupyter/scipy-notebook:2023-12-25
    healthcheck:
      disable: true
```
Hi @hhromic! Thank you for such a detailed issue - you've done a great job investigating this issue.

> If you agree with the above, I can send a PR with this fix.

I do agree and your solution seems to be reasonable.
Also, please, include the test here: https://github.com/jupyter/docker-stacks/blob/main/tests/base-notebook/test_healthcheck.py I think you will only need to add one line of code to this test file.

Resolving patch:
diff --git a/images/base-notebook/docker_healthcheck.py b/images/base-notebook/docker_healthcheck.py
--- a/images/base-notebook/docker_healthcheck.py
+++ b/images/base-notebook/docker_healthcheck.py
@@ -3,6 +3,7 @@
 # Distributed under the terms of the Modified BSD License.
 import json
 import os
+import subprocess
 from pathlib import Path
 
 import requests
@@ -10,7 +11,19 @@
 # Several operations below deliberately don't check for possible errors
 # As this is a healthcheck, it should succeed or raise an exception on error
 
-runtime_dir = Path("/home/") / os.environ["NB_USER"] / ".local/share/jupyter/runtime/"
+# Docker runs healtchecks using an exec
+# It uses the default user configured when running the image: root for the case of a custom NB_USER or jovyan for the case of the default image user.
+# We manually change HOME to make `jupyter --runtime-dir` report a correct path
+# More information: <https://github.com/jupyter/docker-stacks/pull/2074#issuecomment-1879778409>
+result = subprocess.run(
+    ["jupyter", "--runtime-dir"],
+    check=True,
+    capture_output=True,
+    text=True,
+    env=dict(os.environ) | {"HOME": "/home/" + os.environ["NB_USER"]},
+)
+runtime_dir = Path(result.stdout.rstrip())
+
 json_file = next(runtime_dir.glob("*server-*.json"))
 
 url = json.loads(json_file.read_bytes())["url"]


Test patch:
diff --git a/tests/base-notebook/test_healthcheck.py b/tests/base-notebook/test_healthcheck.py
--- a/tests/base-notebook/test_healthcheck.py
+++ b/tests/base-notebook/test_healthcheck.py
@@ -42,6 +42,16 @@
             ["start-notebook.py", "--ServerApp.base_url=/test"],
             "root",
         ),
+        (["JUPYTER_RUNTIME_DIR=/tmp/jupyter-runtime"], ["start-notebook.sh"], None),
+        (
+            [
+                "NB_USER=testuser",
+                "CHOWN_HOME=1",
+                "JUPYTER_RUNTIME_DIR=/tmp/jupyter-runtime",
+            ],
+            ["start-notebook.sh"],
+            "root",
+        ),
     ],
 )
 def test_health(