Repository: sharkdp/fd (language: rust)

Issue:
Binary releases of fd crash on Asahi Linux because of jemalloc using 8kb page sizes
Binary releases crash because they don't support 16kb-aligned pages.

I get the following error with the latest binary releases (both from github releases, and the Arch Linux ARM repos):
```
<jemalloc>: Unsupported system page size
<jemalloc>: Unsupported system page size
memory allocation of 5 bytes failed
[1]    28699 IOT instruction (core dumped)  ./fd-v8.4.0-aarch64-unknown-linux-gnu/fd
```

Building from source fixes the issue (apparently jemalloc has internal logic to detect page size?)

M1 Macs use 16KB pages by default [at a hardware level](https://github.com/AsahiLinux/docs/wiki/Broken-Software#why-not-just-use-4k-pages-then), so this is a common problem with [Ashai Linux](https://github.com/tikv/jemallocator). 

There are two simple fixes that occur to me:
1. disabling jemalloc for AArch64 Linux
2. Compiling jemalloc to use 64K pages (which will work anywhere). This is what the [ALARM Rust](https://github.com/archlinuxarm/PKGBUILDs/commit/19a1393) package does.

The second one will probably have to be filled in the upstream [jemallactor](https://github.com/tikv/jemallocator) project (in fact that could be a more appropriate to file this issue....)


Discussion / hints:
Regarding 1, Asahi/m1 isn't the only way to run linux on aarch64, so disabling jemalloc entirely for the build seems a little extreme. I don't know enough about this to know if there are downsides to 2.

Another option is to build a separate artifact for linux aarch64 without jemalloc and recommend that if you are using asahi.
I think we can configure jemalloc to support 64K pages.  I don't have an easy way to test it though
> I think we can configure jemalloc to support 64K pages. I don't have an easy way to test it though

As I understand it, a jemalloc compiled to use 64K pages supports 8K, 16K, and 64K-aligned processors at runtime (all of which are possibilities on AArch64) . 

Moving the default page size to 64k (or 16k) is probably the best approach. It would keep jemalloc working in the default-compiled binaries.


Increasing the compile-time default is the approach used to fix this problem in most projects:
- [rr debugger](https://github.com/rr-debugger/rr/pull/3146) (switched to 64k)
- [emacs](https://lists.gnu.org/archive/html/bug-gnu-emacs/2021-03/msg01260.html) (switched to 64k)
- [Rust (packaged by Arch Linux ARM)](https://github.com/archlinuxarm/PKGBUILDs/commit/19a1393) (moved the ceiling to 16k)
   - It's not clear what upstream Rust did here 
- [WebKit](https://github.com/rr-debugger/rr/pull/3146) (moved the ceiling to *64k*)

Some projects ([chromium](https://chromium.googlesource.com/chromium/src/+/a704c3a94179e6f3abb28f090d096ced72657d7c), libunwind) switched to runtime detection. jemalloc upstream doesn't currently support that, so we can .

Changing the compile-time default to 64k can be done by overriding the `JEMALLOC_SYS_WITH_LG_PAGE=16` environment variable of the `jemalloc-sys` crate (the variable is in terms of logarithms, and `log2(16 * 1024) = 16)`).

**EDIT**: I'm not sure how we would set this variable using `build.rs`, presumably the `jemallacator` crate is built before the `fd` crate, so `build.rs` couldn't really affect environment variables there.

Maybe we should fix the default upstream?
> **EDIT**: I'm not sure how we would set this variable using `build.rs`, presumably the `jemallacator` crate is built before the `fd` crate, so `build.rs` couldn't really affect environment variables there.

We could just set `JEMALLOC_SYS_WITH_LG_PAGE=16` in the CI environment, and jemallocator should pick it up.  I agree it would be good to report this [upstream](https://github.com/tikv/jemallocator) as well.
I can confirm that setting `JEMALLOC_SYS_WITH_LG_PAGE=16` during a local build fixes the issue for me on Asahi Linux.

It would be great to have the change implemented here so Asahi Linux and other affected `aarch64-linux` users can make use of pre-built binaries. Is something preventing progress on this issue?

> We could just set `JEMALLOC_SYS_WITH_LG_PAGE=16` in the CI environment

If that's all it would take to fix this, I would be willing to open a pull request.
> I would be willing to open a pull request.

@montchr that would be great. 
Would this change be as small as setting a flag in the build step to change the page size?
I was about to make a new issue here, but I think I'm seeing the same thing on a system I'm on? I'm not on Asahi Linux, but Rocky Linux:
```
@$ cat /etc/os-release
NAME="Rocky Linux"
VERSION="9.3 (Blue Onyx)"
...
```
This is on an NVIDIA Grace-Grace node which (I think) uses 64K pages per [this documentation](https://docs.nvidia.com/grace-performance-tuning-guide.pdf). (I have no idea how I'd check this locally...but I'm guessing the integrator used the recommended settings for the most part.)

And when I grab the aarch64 binary and try it out:

```
$ fd cat
<jemalloc>: Unsupported system page size
<jemalloc>: Unsupported system page size
memory allocation of 5 bytes failed
Aborted (core dumped)
```

I'd gladly do more checking or testing if asked...but page sizes and all are a bit beyond my ken. :)

Fix patch:
diff --git a/.github/workflows/CICD.yml b/.github/workflows/CICD.yml
index 7abebfdd2..c07c41cf8 100644
--- a/.github/workflows/CICD.yml
+++ b/.github/workflows/CICD.yml
@@ -134,11 +134,7 @@ jobs:
 
     - name: Build
       shell: bash
-      run: |
-        case ${{ matrix.job.target }} in
-          aarch64-*) export JEMALLOC_SYS_WITH_LG_PAGE=16 ;;
-        esac;
-        $BUILD_CMD build --locked --release --target=${{ matrix.job.target }}
+      run: $BUILD_CMD build --locked --release --target=${{ matrix.job.target }}
 
     - name: Set binary name & path
       id: bin
diff --git a/Cross.toml b/Cross.toml
new file mode 100644
index 000000000..3a146211a
--- /dev/null
+++ b/Cross.toml
@@ -0,0 +1,6 @@
+# https://github.com/sharkdp/fd/issues/1085
+[target.aarch64-unknown-linux-gnu.env]
+passthrough = ["JEMALLOC_SYS_WITH_LG_PAGE=16"]
+
+[target.aarch64-unknown-linux-musl.env]
+passthrough = ["JEMALLOC_SYS_WITH_LG_PAGE=16"]


Test patch: