Repository: static-web-server/static-web-server (language: rust)

Issue:
SWS fails to start on Raspberry Pi 5 with the message "Unsupported system page size"
### Search for duplicate issues

- [X] I already searched, and this issue is not a duplicate.

### Issue scope

Docker / Kubernetes

### Describe the bug

Hi,

  When I try to deploy SWS on a K3S kubernetes server running on a **Raspberry Pi 5**, the containers fail to start with the following error message:
```
<jemalloc>: Unsupported system page size
<jemalloc>: Unsupported system page size
memory allocation of 5 bytes failed
```
Kubernetes information on the container (information anonymized with XXX):
```
kubectl describe pod sws-deployment-66f5b8b9b8-z9vj8
Name:             sws-deployment-66f5b8b9b8-z9vj8
Namespace:        XXXXX
Priority:         0
Service Account:  default
Node:             XXXXX/192.168.XXX.XXX
Start Time:       Sat, 11 May 2024 22:08:49 +0200
Labels:           app=sws
                  pod-template-hash=66f5b8b9b8
Annotations:      <none>
Status:           Running
IP:               10.42.1.26
IPs:
  IP:           10.42.1.26
Controlled By:  ReplicaSet/sws-deployment-66f5b8b9b8
Containers:
  sws:
    Container ID:   containerd://3f227308244481d87f40679cf9c1722654a27ba61e4a65ef90065e3ef1975e6f
    Image:          joseluisq/static-web-server:2.30.0-debian
    Image ID:       docker.io/joseluisq/static-web-server@sha256:c88844fd56f7462c2e0818d49df4dfd3e0ebdb12139e81d72c186df56c97d6a7
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    139
      Started:      Sat, 11 May 2024 22:25:03 +0200
      Finished:     Sat, 11 May 2024 22:25:03 +0200
    Ready:          False
    Restart Count:  8
    Environment:
      FAKE_DISABLE_JEMALLOC:  true
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tmbls (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  kube-api-access-tmbls:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  17m                   default-scheduler  Successfully assigned XXXXX/sws-deployment-66f5b8b9b8-z9vj8 to XXXX
  Normal   Pulling    17m                   kubelet            Pulling image "joseluisq/static-web-server:2.30.0-debian"
  Normal   Pulled     17m                   kubelet            Successfully pulled image "joseluisq/static-web-server:2.30.0-debian" in 6.258s (6.258s including waiting)
  Normal   Pulled     15m (x4 over 17m)     kubelet            Container image "joseluisq/static-web-server:2.30.0-debian" already present on machine
  Normal   Created    15m (x5 over 17m)     kubelet            Created container sws
  Normal   Started    15m (x5 over 17m)     kubelet            Started container sws
  Warning  BackOff    2m34s (x69 over 17m)  kubelet            Back-off restarting failed container sws in pod sws-deployment-66f5b8b9b8-z9vj8_XXXXX(d6040825-8420-4f32-9fc2-52389ab1c7db)
  ```
  
  The same error is present with the _scratch_ version or the docker image.

  I found a bug that has been fixed and could be related to the same root cause on another software: https://github.com/qdrant/qdrant/issues/2474 or [another related PR](https://github.com/risingwavelabs/risingwave/pull/15956/commits/86d91397cfa45b6faf74f8c4926ea307380c8654) and also https://github.com/typesense/typesense/issues/1351 . I tried to add the _DISABLE_JEMALLOC_ environment variable but it did not fix the issue (Note: then I renamed it to avoid potential conflicts).

For information, the page size seems to be configured to 16K on the system (total of 8Gi of RAM installed).
```
getconf PAGE_SIZE                           
16384
```
Have a nice day.


### How to reproduce it

1. Install a k3s server on a Rasberry Pi 5B with 8G of RAM.

2. Deploy SWS using the following deployment file name _sws.yaml_:
```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sws-deployment
  labels:
    app: sws
spec:
  replicas: 2
  selector:
    matchLabels:
      app: sws
  template:
    metadata:
      labels:
        app: sws
    spec:
      containers:
      - name: sws
        image: joseluisq/static-web-server:2.30.0-debian
        ports:
        - containerPort: 80
```
Command: `kubectl apply -f sws.yaml`

3. Check the pods status with `kubectl get pods`
```
NAME                              READY   STATUS             RESTARTS         AGE
sws-deployment-66f5b8b9b8-z9vj8   0/1     CrashLoopBackOff   10 (3m41s ago)   30m
sws-deployment-66f5b8b9b8-644jw   0/1     CrashLoopBackOff   10 (3m34s ago)   30m
```

The pods should be in the _Running_ state but are in the _CrashLoopBackOff_ after a few minutes.

The logs are indicating an issue with _jemalloc_ : `kubectl logs sws-deployment-66f5b8b9b8-z9vj8`

```
<jemalloc>: Unsupported system page size
<jemalloc>: Unsupported system page size
memory allocation of 5 bytes failed
```


### Expected behavior

The pods should be in the _Running_ state after the deployment and not error should be logged.

### Complementary information

The image deployed seem to be the correct one as the SHA matches the k3s description above: https://hub.docker.com/layers/joseluisq/static-web-server/2.30.0-debian/images/sha256-b7dce340af9f5d282de917dd9ae27306e0620f54377da4b0ce51cf32f1538329?context=explore

The issue seems to be documented on the Raspberry Pi issue tracker: https://github.com/raspberrypi/bookworm-feedback/issues/107

### Build target

Docker ~linux/amd64~ linux/arm64

### Environment and specs

- [X] **static-web-server:** 2.30.0-debian
- [x] **OS:** Linux XXXXX 6.6.28+rpt-rpi-2712 # 1 SMP PREEMPT Debian 1:6.6.28-1+rpt1 (2024-04-22) aarch64 GNU/Linux
- [x] **Arch:** ARM64 (64-bit) on Rasberry Pi 5B
- [x] **Docker:** k3s kubernetes version v1.29.4+k3s1
- [x] **Client:** Not applicable
- [ ] Specify others


### Additional context

_No response_


Discussion / hints:
# Workaround

After some web search, i found a workaround that could confirm the issue with the page 16K page size ( https://github.com/raspberrypi/bookworm-feedback/issues/107 ). I added the `kernel=kernel8.img` line after the `[all]` section in `/boot/firmware/config.txt` file and rebooted the Raspberry Pi 5.

```
[all]

kernel=kernel8.img
```

Now `uname -a` show that the kernel type/version changed from _2712_ to _8_ :
```
Linux XXXX 6.6.28+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.28-1+rpt1 (2024-04-22) aarch64
```

The pods are in the _Running_ state:
```
NAME                              READY   STATUS    RESTARTS        AGE
sws-deployment-66f5b8b9b8-z9vj8   1/1     Running   111 (16m ago)   9h
sws-deployment-66f5b8b9b8-644jw   1/1     Running   111 (15m ago)   9h
```
And the logs show a successful start of SWS:
```
2024-05-12T05:10:05.220487Z  INFO static_web_server::info: static-web-server 2.30.0
2024-05-12T05:10:05.220524Z  INFO static_web_server::info: log level: error
2024-05-12T05:10:05.220596Z  INFO static_web_server::info: server bound to tcp socket [::]:80
2024-05-12T05:10:05.220600Z  INFO static_web_server::info: runtime worker threads: 4
2024-05-12T05:10:05.220603Z  INFO static_web_server::info: runtime max blocking threads: 512
2024-05-12T05:10:05.228592Z  INFO static_web_server::info: log remote address: enabled=false
2024-05-12T05:10:05.228606Z  INFO static_web_server::info: redirect trailing slash: enabled=true
2024-05-12T05:10:05.228611Z  INFO static_web_server::info: ignore hidden files: enabled=false
2024-05-12T05:10:05.228614Z  INFO static_web_server::info: grace period before graceful shutdown: 0s
2024-05-12T05:10:05.228619Z  INFO static_web_server::info: index files: index.html
2024-05-12T05:10:05.228623Z  INFO static_web_server::info: directory listing: enabled=false
2024-05-12T05:10:05.228631Z  INFO static_web_server::info: directory listing order code: 6
2024-05-12T05:10:05.228634Z  INFO static_web_server::info: directory listing format: Html
2024-05-12T05:10:05.237242Z  INFO static_web_server::info: fallback page: enabled=false, value=""
2024-05-12T05:10:05.237256Z  INFO static_web_server::info: health endpoint: enabled=false
2024-05-12T05:10:05.237259Z  INFO static_web_server::info: metrics endpoint (experimental): enabled=false
2024-05-12T05:10:05.237266Z  INFO static_web_server::info: basic authentication: enabled=false
2024-05-12T05:10:05.237269Z  INFO static_web_server::info: maintenance mode: enabled=false
2024-05-12T05:10:05.237272Z  INFO static_web_server::info: maintenance mode status: 503
2024-05-12T05:10:05.237275Z  INFO static_web_server::info: maintenance mode file: ""
2024-05-12T05:10:05.237278Z  INFO static_web_server::info: compression static: enabled=false
2024-05-12T05:10:05.237283Z  INFO static_web_server::info: auto compression: enabled=true, formats=deflate,gzip,brotli,zstd
2024-05-12T05:10:05.237286Z  INFO static_web_server::info: cache control headers: enabled=true
2024-05-12T05:10:05.237289Z  INFO static_web_server::info: security headers: enabled=false
2024-05-12T05:10:05.252081Z  INFO static_web_server::info: http1 server is listening on http://[::]:80
2024-05-12T05:10:05.252091Z  INFO static_web_server::info: press ctrl+c to shut down the server
```
I wonder which OS/Arch musl-binary Docker is choosing for your PI 5 in Kubernetes.
Regardless, you could try to build a new Docker image but using the dynamic-linked binary `static-web-server-v2.30.0-aarch64-unknown-linux-gnu.tar.gz` instead of the static one `static-web-server-v2.30.0-aarch64-unknown-linux-musl.tar.gz` (currently published).


What I'm also thinking is to provide dynamic-linked binaries for all Docker Debian-based images instead of using musl and leaving musl binaries only for the Alpine images. This should avoid headaches like this.


@joseluisq the OS/Arch musl-binary chosen for the PI 5 in Kubernetes seems to be `joseluisq/static-web-server:2.30.0` images for `linux/arm64`.

A `kubectl describe pod` commands returns the following information `Image ID:       docker.io/joseluisq/static-web-server@sha256:4a878e15395678168c20acdf0daf7381e5b687436da06821f52c8b786582571e` which matches the sha of the image on [Docker Hub](https://hub.docker.com/layers/joseluisq/static-web-server/2.30.0/images/sha256-b31e3123b31de9a5547dfae2fd694ae32dac92258f1f787a784aefefbbd06ece?context=explore)

Please note that with the workaround in place I switched back to the _Scratch_ version of the image.


> @joseluisq the OS/Arch musl-binary chosen for the PI 5 in Kubernetes seems to be `joseluisq/static-web-server:2.30.0` images for `linux/arm64`.
> 
> A `kubectl describe pod` commands returns the following information `Image ID: docker.io/joseluisq/static-web-server@sha256:4a878e15395678168c20acdf0daf7381e5b687436da06821f52c8b786582571e` which matches the sha of the image on [Docker Hub](https://hub.docker.com/layers/joseluisq/static-web-server/2.30.0/images/sha256-b31e3123b31de9a5547dfae2fd694ae32dac92258f1f787a784aefefbbd06ece?context=explore)

Ok, then It is using the `aarch64-unknown-linux-musl` binary.

@fred00 Can you please build a Debian image with the [aarch64-unknown-linux-gnu](https://github.com/static-web-server/static-web-server/releases/download/v2.30.0/static-web-server-v2.30.0-aarch64-unknown-linux-gnu.tar.gz) binary and test it in your PI 5 with Kubernetes? Let me know 
if that worked.
I have the same issue on my Raspberry Pi 5. I have freshly installed Raspberry Pi OS with default settings on my Pi (the page size of 16kb is the default). According to Raspberry Pi they changed the page size to improve performance. Would be a bummer to not use this performance improvements.
Thank you all for working on a solution 👍🏻 

Can some of you @sharknoon or @fred00 build a Debian image with the [aarch64-unknown-linux-gnu](https://github.com/static-web-server/static-web-server/releases/download/v2.30.0/static-web-server-v2.30.0-aarch64-unknown-linux-gnu.tar.gz) binary and test it if works fine?

The issue is about jemalloc but it is only used in musl static-binaries like `aarch64-unknown-linux-musl`.
Will do so when I am home (in about 8 hours) 👍
Ok, exploring a bit more in detail. Turns out that it is a well-known problem with a solution.
Basically, we can keep musl static/linked binaries for `aarch64` (a.k.a. ARM64) targets but we have to tell jemalloc to use a larger page size by default.

So we can go with using the maximal supported page size of `64KB` on `aarch64` which should be not a problem for system with smaller page sizes since jemalloc supports that functionality. See https://github.com/jemalloc/jemalloc/pull/769

I will prepare a fix soon.
@joseluisq I’m not sure to have the toolchain required to build a new image for aarch64. If @sharknoon can’t do it I will try my best. 
If you prepare an image with a fix, I will be able to test it with both Pi 5 kernels to provide you with a feedback on the fix compatibility. 
@joseluisq It this how you imagined the Debian test image? [Dockerfile](https://github.com/sharknoon/tmp/blob/main/Dockerfile)
Quick Update: I have managed to get a Raspberry Pi 5 at work and it works fine with the binaries that @joseluisq has provided
[here](https://github.com/static-web-server/static-web-server/releases/download/v2.30.0/static-web-server-v2.30.0-aarch64-unknown-linux-gnu.tar.gz)
> Quick Update: I have managed to get a Raspberry Pi 5 at work and it works fine with the binaries that @joseluisq has provided [here](https://github.com/static-web-server/static-web-server/releases/download/v2.30.0/static-web-server-v2.30.0-aarch64-unknown-linux-gnu.tar.gz)

Thanks for checking. That's what I was assuming. I just wanted to double-check.

Now, two things to clarify.

First, regarding the musl static-linked binary for `linux/arm64`, we still want to compile jemalloc with a 64KB page size default.

Second, regarding the Debian Docker image for `linux/arm64`, I'm still thinking on why not provide dynamic-linked binaries (glibc) like `aarch64-unknown-linux-gnu` for the Debian-based images only and leave the static-linked binaries (musl) for the Alpine and the Scratch images for consistency. This has to come up later at some point. But for now, we will fix the issue by increasing the page size as discussed in https://github.com/static-web-server/static-web-server/issues/440#issuecomment-2106958346.
Thank you! I am really amazed how quickly you responded to this error. The static web server is an amazing software and I really enjoy using it. Thank you for your time and effort, I really appreciate it 😃
@joseluisq, just to confirm @sharknoon feedback, I started the binaries directly on the Pi 5 (outside of Docker/K8S) on both kernels and it starts successfully on them. Thank you very much for your exceptional support!

```
$ uname -a
Linux XXX 6.6.28+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.28-1+rpt1 (2024-04-22) aarch64 GNU/Linux

$ ./static-web-server -p 1234 
2024-05-13T16:55:50.293000Z  INFO static_web_server::info: static-web-server 2.30.0
2024-05-13T16:55:50.293030Z  INFO static_web_server::info: log level: error
2024-05-13T16:55:50.293071Z  INFO static_web_server::info: server bound to tcp socket [::]:1234
2024-05-13T16:55:50.293078Z  INFO static_web_server::info: runtime worker threads: 4
2024-05-13T16:55:50.293082Z  INFO static_web_server::info: runtime max blocking threads: 512
2024-05-13T16:55:50.293097Z  INFO static_web_server::info: log remote address: enabled=false
2024-05-13T16:55:50.293102Z  INFO static_web_server::info: redirect trailing slash: enabled=true
2024-05-13T16:55:50.293107Z  INFO static_web_server::info: ignore hidden files: enabled=false
2024-05-13T16:55:50.293111Z  INFO static_web_server::info: grace period before graceful shutdown: 0s
2024-05-13T16:55:50.293119Z  INFO static_web_server::info: index files: index.html
2024-05-13T16:55:50.293124Z  INFO static_web_server::info: directory listing: enabled=false
2024-05-13T16:55:50.293129Z  INFO static_web_server::info: directory listing order code: 6
2024-05-13T16:55:50.293133Z  INFO static_web_server::info: directory listing format: Html
2024-05-13T16:55:50.293141Z  INFO static_web_server::info: fallback page: enabled=false, value=""
2024-05-13T16:55:50.293146Z  INFO static_web_server::info: health endpoint: enabled=false
2024-05-13T16:55:50.293150Z  INFO static_web_server::info: metrics endpoint (experimental): enabled=false
2024-05-13T16:55:50.293156Z  INFO static_web_server::info: basic authentication: enabled=false
2024-05-13T16:55:50.293160Z  INFO static_web_server::info: maintenance mode: enabled=false
2024-05-13T16:55:50.293164Z  INFO static_web_server::info: maintenance mode status: 503
2024-05-13T16:55:50.293169Z  INFO static_web_server::info: maintenance mode file: ""
2024-05-13T16:55:50.293173Z  INFO static_web_server::info: compression static: enabled=false
2024-05-13T16:55:50.293177Z  INFO static_web_server::info: auto compression: enabled=true, formats=deflate,gzip,brotli,zstd
2024-05-13T16:55:50.293182Z  INFO static_web_server::info: cache control headers: enabled=true
2024-05-13T16:55:50.293186Z  INFO static_web_server::info: security headers: enabled=false
2024-05-13T16:55:50.293242Z  INFO static_web_server::info: http1 server is listening on http://[::]:1234
2024-05-13T16:55:50.293259Z  INFO static_web_server::info: press ctrl+c to shut down the server
```

```
uname -a
Linux XXX 6.6.28+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.28-1+rpt1 (2024-04-22) aarch64 GNU/Linux

$ ./static-web-server -p 1234 
2024-05-13T16:51:05.036837Z  INFO static_web_server::info: static-web-server 2.30.0
2024-05-13T16:51:05.036865Z  INFO static_web_server::info: log level: error
2024-05-13T16:51:05.036904Z  INFO static_web_server::info: server bound to tcp socket [::]:1234
2024-05-13T16:51:05.036910Z  INFO static_web_server::info: runtime worker threads: 4
2024-05-13T16:51:05.036914Z  INFO static_web_server::info: runtime max blocking threads: 512
2024-05-13T16:51:05.036930Z  INFO static_web_server::info: log remote address: enabled=false
2024-05-13T16:51:05.036935Z  INFO static_web_server::info: redirect trailing slash: enabled=true
2024-05-13T16:51:05.036940Z  INFO static_web_server::info: ignore hidden files: enabled=false
2024-05-13T16:51:05.036944Z  INFO static_web_server::info: grace period before graceful shutdown: 0s
2024-05-13T16:51:05.036954Z  INFO static_web_server::info: index files: index.html
2024-05-13T16:51:05.036960Z  INFO static_web_server::info: directory listing: enabled=false
2024-05-13T16:51:05.036964Z  INFO static_web_server::info: directory listing order code: 6
2024-05-13T16:51:05.036969Z  INFO static_web_server::info: directory listing format: Html
2024-05-13T16:51:05.036979Z  INFO static_web_server::info: fallback page: enabled=false, value=""
2024-05-13T16:51:05.036984Z  INFO static_web_server::info: health endpoint: enabled=false
2024-05-13T16:51:05.036989Z  INFO static_web_server::info: metrics endpoint (experimental): enabled=false
2024-05-13T16:51:05.036995Z  INFO static_web_server::info: basic authentication: enabled=false
2024-05-13T16:51:05.036999Z  INFO static_web_server::info: maintenance mode: enabled=false
2024-05-13T16:51:05.037003Z  INFO static_web_server::info: maintenance mode status: 503
2024-05-13T16:51:05.037007Z  INFO static_web_server::info: maintenance mode file: ""
2024-05-13T16:51:05.037011Z  INFO static_web_server::info: compression static: enabled=false
2024-05-13T16:51:05.037016Z  INFO static_web_server::info: auto compression: enabled=true, formats=deflate,gzip,brotli,zstd
2024-05-13T16:51:05.037022Z  INFO static_web_server::info: cache control headers: enabled=true
2024-05-13T16:51:05.037026Z  INFO static_web_server::info: security headers: enabled=false
2024-05-13T16:51:05.037095Z  INFO static_web_server::info: http1 server is listening on http://[::]:1234
2024-05-13T16:51:05.037106Z  INFO static_web_server::info: press ctrl+c to shut down the server
```



Fix patch:
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
index 4ec61d8d..a22bf038 100644
--- a/.github/workflows/release.yml
+++ b/.github/workflows/release.yml
@@ -85,6 +85,9 @@ jobs:
           os: ubuntu-22.04
           rust: stable
           target: aarch64-unknown-linux-musl
+          # jemalloc 64KB (16) pagesize by default
+          # See: https://github.com/jemalloc/jemalloc/issues/467
+          jemalloc_sys_with_lg_page: 16
         - build: linux-gnu
           os: ubuntu-22.04
           rust: stable
@@ -209,6 +212,11 @@ jobs:
         # ring crate: add Visual Studio Build Tools "VS 2022 C++ ARM64 build tools" and "clang" components
         $env:Path += ";C:\Program Files (x86)\Microsoft Visual Studio\2022\Enterprise\VC\Tools\Llvm\x64\bin"
 
+    - name: Setup Linux ARM64 MUSL
+      if: ${{ contains(matrix.build, 'linux-musl-arm64') }}
+      run: |
+        echo "JEMALLOC_SYS_WITH_LG_PAGE=${{ matrix.jemalloc_sys_with_lg_page }}" >> $GITHUB_ENV
+
     - name: Show command used for Cargo
       run: |
         echo "cargo command is: ${{ env.CARGO_BIN }}"


Test patch: