Repository: jupyter/docker-stacks

Issue:
[BUG] - Health Check fails if you change the port jupyter runs on
### What docker image(s) are you using?

minimal-notebook

### OS system and architecture running docker image

RHEL7  docker swarm

### What Docker command are you running?

Not really relevant, but I need to run it in a docker swarm, with a generalise 'ingress service'.

For this I needed to change internal port jupyter runs on needs to be changes for intergation into a 'ingress proxy'
To change the port I made a slight modification the docker image to set the internal port it runs on (see below)

The problem is the docker container  dies unexpectedly after running for 46 seconds.
During that time the service is visible within the conatiner, but not external to the container.
This is because the built-in heathcheck never succeeds, and eventually kills the container with little logged reporting.
(see below)


### How to Reproduce the problem?

Dockerfile, to set port

```dockerfile
FROM "jupyter/minimal-notebook:latest"
# Update Jupyter configuration to set port
RUN set -eux; \
    sed -i 's/port = 8888/port = 8080/' /etc/jupyter/jupyter_notebook_config.py ;\
    sed -i 's/port = 8888/port = 8080/' /etc/jupyter/jupyter_server_config.py ;\
    :;
```
You can also change the port in other ways such as... 

Creating a `~joyvan/.jupyter/jupyter_server_config.py` file (which can also set a password)

or setting a  JUPYTER_PORT environment variable (IF the setting in `/etc/jupyter` configs are removed)



### Command output

When you build and then run the modified docker image, `docker ps`  reports
`Up 9 seconds (health: starting)   8888/tcp`  despite the fact that jupyter is now running on port 8080

46 seconds after starting the container dies with a unhelpful (Signal 15)

Log output...
```
[I 2022-10-28 05:20:00.393 ServerApp] Jupyter Server 1.21.0 is running at:
[I 2022-10-28 05:20:00.393 ServerApp] http://jupyter_service:8080/lab?token=929eaaf2e60f8a947761cb1f2741d745fd46dde62f6fef7c
or http://127.0.0.1:8080/lab?token=929eaaf2e60f8a947761cb1f2741d745fd46dde62f6fef7c
[I 2022-10-28 05:20:00.393 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 2022-10-28 05:20:00.397 ServerApp] 

To access the server, open this file in a browser:
file:///home/jovyan/.local/share/jupyter/runtime/jpserver-8-open.html
Or copy and paste one of these URLs:
http://jupyter_service:8080/lab?token=929eaaf2e60f8a947761cb1f2741d745fd46dde62f6fef7c
or http://127.0.0.1:8080/lab?token=929eaaf2e60f8a947761cb1f2741d745fd46dde62f6fef7c
Entered start.sh with args: jupyter lab
Executing the command: jupyter lab
[C 2022-10-28 05:20:46.261 ServerApp] received signal 15, stopping
[I 2022-10-28 05:20:46.262 ServerApp] Shutting down 2 extensions
[I 2022-10-28 05:20:46.263 ServerApp] Shutting down 0 terminals
```

### Expected behavior

Changing the internal port should not take days of work to track down, it should be straight forward and documented.

The healthcheck should also be properly documented in jupyter-stacks documentation.
This will make it more 'swarm friendly' as well as allow others to integrate it better when port 8888 is NOT available.

Yes you can map the port when doing a 'docker run', but that is NOT always possible.

### Actual behavior

Internal Port changing is undocumented in stacks

Heathcheck kills the container without notice (signal 15 hardly makes it clear) when port is different.

Days of work lost trying to figure out what should be a straight forward and simple task.

### Anything else?

There is an existing environment variable  "JUPYTER_PORT" that defines the default port.
But any such setting is currently overridden by the  configuration files in `/etc/jupyter`

This may be usable to set healthcheck, especially if the config file default is removed, or allows the env var to override.

in Dockerfile....
```
HEALTHCHECK  --interval=15s --timeout=3s --start-period=5s --retries=3 \
    CMD wget -O- --no-verbose --tries=1 --no-check-certificate \
    http${GEN_CERT:+s}://localhost:${JUPYTER_PORT:-8888}${JUPYTERHUB_SERVICE_PREFIX:-/}api || exit 1
```

That Environment variable also needs to be documented in the jupyter-stacks documentation, with the health check.
[BUG] - Health Check fails if you change the port jupyter runs on
### What docker image(s) are you using?

minimal-notebook

### OS system and architecture running docker image

RHEL7  docker swarm

### What Docker command are you running?

Not really relevant, but I need to run it in a docker swarm, with a generalise 'ingress service'.

For this I needed to change internal port jupyter runs on needs to be changes for intergation into a 'ingress proxy'
To change the port I made a slight modification the docker image to set the internal port it runs on (see below)

The problem is the docker container  dies unexpectedly after running for 46 seconds.
During that time the service is visible within the conatiner, but not external to the container.
This is because the built-in heathcheck never succeeds, and eventually kills the container with little logged reporting.
(see below)


### How to Reproduce the problem?

Dockerfile, to set port

```dockerfile
FROM "jupyter/minimal-notebook:latest"
# Update Jupyter configuration to set port
RUN set -eux; \
    sed -i 's/port = 8888/port = 8080/' /etc/jupyter/jupyter_notebook_config.py ;\
    sed -i 's/port = 8888/port = 8080/' /etc/jupyter/jupyter_server_config.py ;\
    :;
```
You can also change the port in other ways such as... 

Creating a `~joyvan/.jupyter/jupyter_server_config.py` file (which can also set a password)

or setting a  JUPYTER_PORT environment variable (IF the setting in `/etc/jupyter` configs are removed)



### Command output

When you build and then run the modified docker image, `docker ps`  reports
`Up 9 seconds (health: starting)   8888/tcp`  despite the fact that jupyter is now running on port 8080

46 seconds after starting the container dies with a unhelpful (Signal 15)

Log output...
```
[I 2022-10-28 05:20:00.393 ServerApp] Jupyter Server 1.21.0 is running at:
[I 2022-10-28 05:20:00.393 ServerApp] http://jupyter_service:8080/lab?token=929eaaf2e60f8a947761cb1f2741d745fd46dde62f6fef7c
or http://127.0.0.1:8080/lab?token=929eaaf2e60f8a947761cb1f2741d745fd46dde62f6fef7c
[I 2022-10-28 05:20:00.393 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 2022-10-28 05:20:00.397 ServerApp] 

To access the server, open this file in a browser:
file:///home/jovyan/.local/share/jupyter/runtime/jpserver-8-open.html
Or copy and paste one of these URLs:
http://jupyter_service:8080/lab?token=929eaaf2e60f8a947761cb1f2741d745fd46dde62f6fef7c
or http://127.0.0.1:8080/lab?token=929eaaf2e60f8a947761cb1f2741d745fd46dde62f6fef7c
Entered start.sh with args: jupyter lab
Executing the command: jupyter lab
[C 2022-10-28 05:20:46.261 ServerApp] received signal 15, stopping
[I 2022-10-28 05:20:46.262 ServerApp] Shutting down 2 extensions
[I 2022-10-28 05:20:46.263 ServerApp] Shutting down 0 terminals
```

### Expected behavior

Changing the internal port should not take days of work to track down, it should be straight forward and documented.

The healthcheck should also be properly documented in jupyter-stacks documentation.
This will make it more 'swarm friendly' as well as allow others to integrate it better when port 8888 is NOT available.

Yes you can map the port when doing a 'docker run', but that is NOT always possible.

### Actual behavior

Internal Port changing is undocumented in stacks

Heathcheck kills the container without notice (signal 15 hardly makes it clear) when port is different.

Days of work lost trying to figure out what should be a straight forward and simple task.

### Anything else?

There is an existing environment variable  "JUPYTER_PORT" that defines the default port.
But any such setting is currently overridden by the  configuration files in `/etc/jupyter`

This may be usable to set healthcheck, especially if the config file default is removed, or allows the env var to override.

in Dockerfile....
```
HEALTHCHECK  --interval=15s --timeout=3s --start-period=5s --retries=3 \
    CMD wget -O- --no-verbose --tries=1 --no-check-certificate \
    http${GEN_CERT:+s}://localhost:${JUPYTER_PORT:-8888}${JUPYTERHUB_SERVICE_PREFIX:-/}api || exit 1
```

That Environment variable also needs to be documented in the jupyter-stacks documentation, with the health check.


Discussion:
+1, I also experienced similar issue. In my case, I used custom port as well as changing the base url (using the `--ServerApp.base_url` parameter) as I want to host jupyter from a subpath.

The workaround that I came up with is to override the healthcheck using a custom-build downstream docker image or by  specifying healthcheck on docker compose configuration.

This should be better documented, and would be even better if there is easier way to specify custom ports and base url, just as how we can specify JUPYTERHUB_SERVICE_PREFIX using environment variable. 
Using JUPYTER_PORT (sets default port used) would be ideal. but needs the default port setting removed from the /etc config files (or it overrides it) and the variable added to the HEALTHCHECK.  Both of which substitute the environment variable value at run time, rather than at the time of Dockerfile image build.  

So if it is fixed in the library docker images, you can set it at run time, just like JUPYTERHUB_SERVICE_PREFIX 
Hi everyone, thank you for all your suggestions!
`JUPYTER_PORT` was not there when these stacks were created, so I think it's a great idea to switch from config to env variable. https://github.com/jupyter/notebook/pull/5221

Is someone willing to create a PR for this?
1. Let's remove port setting in config.
2. Add `JUPYTER_PORT` with default value `8888` to the root file
3. Add a test which changes the internal port
4. Fix the health check command to use this env variable
5. Add a health check test (both using the default port and not using it).
6. Document health check behaviour (including https://github.com/jupyter/docker-stacks/issues/1709)
@antofthy @imiro any chance you could work on a PR?
Sorry I've been a bit busy, but I might have some time next week on this
@imiro if you're still interested, I've recently added a new healthcheck test.
+1, I also experienced similar issue. In my case, I used custom port as well as changing the base url (using the `--ServerApp.base_url` parameter) as I want to host jupyter from a subpath.

The workaround that I came up with is to override the healthcheck using a custom-build downstream docker image or by  specifying healthcheck on docker compose configuration.

This should be better documented, and would be even better if there is easier way to specify custom ports and base url, just as how we can specify JUPYTERHUB_SERVICE_PREFIX using environment variable. 
Using JUPYTER_PORT (sets default port used) would be ideal. but needs the default port setting removed from the /etc config files (or it overrides it) and the variable added to the HEALTHCHECK.  Both of which substitute the environment variable value at run time, rather than at the time of Dockerfile image build.  

So if it is fixed in the library docker images, you can set it at run time, just like JUPYTERHUB_SERVICE_PREFIX 
Hi everyone, thank you for all your suggestions!
`JUPYTER_PORT` was not there when these stacks were created, so I think it's a great idea to switch from config to env variable. https://github.com/jupyter/notebook/pull/5221

Is someone willing to create a PR for this?
1. Let's remove port setting in config.
2. Add `JUPYTER_PORT` with default value `8888` to the root file
3. Add a test which changes the internal port
4. Fix the health check command to use this env variable
5. Add a health check test (both using the default port and not using it).
6. Document health check behaviour (including https://github.com/jupyter/docker-stacks/issues/1709)
@antofthy @imiro any chance you could work on a PR?
Sorry I've been a bit busy, but I might have some time next week on this
@imiro if you're still interested, I've recently added a new healthcheck test.

Resolving patch:
diff --git a/base-notebook/jupyter_server_config.py b/base-notebook/jupyter_server_config.py
--- a/base-notebook/jupyter_server_config.py
+++ b/base-notebook/jupyter_server_config.py
@@ -9,7 +9,6 @@
 
 c = get_config()  # noqa: F821
 c.ServerApp.ip = "0.0.0.0"
-c.ServerApp.port = 8888
 c.ServerApp.open_browser = False
 
 # to output both image/svg+xml and application/pdf plot formats in the notebook file


Test patch:
diff --git a/tests/base-notebook/test_container_options.py b/tests/base-notebook/test_container_options.py
--- a/tests/base-notebook/test_container_options.py
+++ b/tests/base-notebook/test_container_options.py
@@ -78,3 +78,39 @@ def test_unsigned_ssl(
     assert "ERROR" not in logs
     warnings = TrackedContainer.get_warnings(logs)
     assert not warnings
+
+
+@pytest.mark.parametrize(
+    "env",
+    [
+        {},
+        {"JUPYTER_PORT": 1234, "DOCKER_STACKS_JUPYTER_CMD": "lab"},
+        {"JUPYTER_PORT": 2345, "DOCKER_STACKS_JUPYTER_CMD": "notebook"},
+        {"JUPYTER_PORT": 3456, "DOCKER_STACKS_JUPYTER_CMD": "server"},
+        {"JUPYTER_PORT": 4567, "DOCKER_STACKS_JUPYTER_CMD": "nbclassic"},
+        {"JUPYTER_PORT": 5678, "RESTARTABLE": "yes"},
+        {"JUPYTER_PORT": 6789},
+        {"JUPYTER_PORT": 7890, "DOCKER_STACKS_JUPYTER_CMD": "notebook"},
+    ],
+)
+def test_custom_internal_port(
+    container: TrackedContainer,
+    http_client: requests.Session,
+    env: dict[str, str],
+) -> None:
+    """Container should be accessible from the host
+    when using custom internal port"""
+    host_port = find_free_port()
+    internal_port = env.get("JUPYTER_PORT", 8888)
+    running_container = container.run_detached(
+        command=["start-notebook.sh", "--NotebookApp.token=''"],
+        environment=env,
+        ports={internal_port: host_port},
+    )
+    resp = http_client.get(f"http://localhost:{host_port}")
+    resp.raise_for_status()
+    logs = running_container.logs().decode("utf-8")
+    LOGGER.debug(logs)
+    assert "ERROR" not in logs
+    warnings = TrackedContainer.get_warnings(logs)
+    assert not warnings
diff --git a/tests/base-notebook/test_healthcheck.py b/tests/base-notebook/test_healthcheck.py
--- a/tests/base-notebook/test_healthcheck.py
+++ b/tests/base-notebook/test_healthcheck.py
@@ -17,10 +17,12 @@
     [
         None,
         ["DOCKER_STACKS_JUPYTER_CMD=lab"],
-        ["RESTARTABLE=yes"],
         ["DOCKER_STACKS_JUPYTER_CMD=notebook"],
         ["DOCKER_STACKS_JUPYTER_CMD=server"],
         ["DOCKER_STACKS_JUPYTER_CMD=nbclassic"],
+        ["RESTARTABLE=yes"],
+        ["JUPYTER_PORT=8171"],
+        ["JUPYTER_PORT=8117", "DOCKER_STACKS_JUPYTER_CMD=notebook"],
     ],
 )
 def test_health(container: TrackedContainer, env: Optional[list[str]]) -> None: