HelloSun/fint-epqv2

Flint EPQ for llama.cpp

CPU expert paging for unsloth/flint-GGUF (flint-UD-Q4_K_XL.gguf). Routed expert weights stay on disk and are loaded into a RAM arena on demand. Full results: docs/REPORT.md.

Install

Requires a CMake build of llama.cpp at commit 970b51ce7c9ba5fff0726c5e9994f1c2f0e4eedc with flint-epq.patch applied (sha256 369ef9710e507f5f8caeae398382399b43dd9adeeb006dfd27ce5e84bc041db4).

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout 970b51ce7c9ba5fff0726c5e9994f1c2f0e4eedc
curl -L 'https://huggingface.co/HelloSun/fint-epqv2/resolve/main/flint-epq.patch' -o flint-epq.patch
git apply flint-epq.patch
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF -DGGML_NATIVE=OFF \
  -DGGML_AVX2=ON -DGGML_AVX512=ON -DGGML_AVX512_VBMI=ON -DGGML_AVX512_VNNI=ON -DGGML_AVX512_BF16=ON \
  -DGGML_AMX_INT8=OFF -DGGML_AMX_TILE=OFF -DGGML_AMX_BF16=OFF
cmake --build build -j 8 --target llama-cli

hf download unsloth/flint-GGUF flint-UD-Q4_K_XL.gguf --local-dir models

Run

Examples use the 4 GiB memory tier (SDQ_ARENA_MB=1024, a 1 GiB expert arena). Prompt: Explain in one sentence why the sky is blue.

native (unmodified llama.cpp MoE path, needs about 24 GiB RAM, so it does not fit in 4 GiB):

SDQ_PAGER=0 build/bin/llama-cli -m models/flint-UD-Q4_K_XL.gguf -c 512 \
  -p 'Explain in one sentence why the sky is blue.' -n 32 -t 16 --temp 0 -st --simple-io

v1 (expert paging, mode C). Preload and row-parallel default to on, so turn them off explicitly:

SDQ_PAGER=1 SDQ_ARENA_MB=1024 SDQ_PRELOAD=0 SDQ_ROW_PARALLEL=0 \
  build/bin/llama-cli -m models/flint-UD-Q4_K_XL.gguf -c 512 \
  -p 'Explain in one sentence why the sky is blue.' -n 32 -t 16 --temp 0 -st --simple-io

v2 (default settings, preload and row-parallel on). At 4 GiB the arena cannot hold all experts, so v2 runs the same path as v1 here:

SDQ_PAGER=1 SDQ_ARENA_MB=1024 \
  build/bin/llama-cli -m models/flint-UD-Q4_K_XL.gguf -c 512 \
  -p 'Explain in one sentence why the sky is blue.' -n 32 -t 16 --temp 0 -st --simple-io

auto (no SDQ_ARENA_MB): the arena is 80 % of available memory, capped at the size of all experts. Available memory is the smaller of MemAvailable and the cgroup headroom (memory.max - memory.current). With at least about 36 GiB available the cap is reached, so preload and row-parallel turn on automatically (the same behavior as v2):

SDQ_PAGER=1 build/bin/llama-cli -m models/flint-UD-Q4_K_XL.gguf -c 512 \
  -p 'Explain in one sentence why the sky is blue.' -n 32 -t 16 --temp 0 -st --simple-io

The remaining 20 % must cover the non-expert weights and compute buffers, which load after the arena is sized. The auto estimate uses the cgroup's current usage, including reclaimable page cache, so it is conservative. Small limits have not been tested in auto mode; set SDQ_ARENA_MB explicitly there.

v2 changes behavior only when the arena holds all 12,032 experts (≥ 28.6 GiB, for example SDQ_ARENA_MB=29400).

Use 16 threads (-t 16). Row-parallel needs more threads than the number of experts used per step (6), so at least 7.

Notes

• Memory at 4 GiB tier: peak 2.76 GiB. The process needs about 1.8 GiB plus the arena. Under a 2 GiB limit a 128 MiB arena works (peak 1.89 GiB); 256 MiB does not.
• Generation speed, 4 GiB tier, 16 threads (median): v1 4.6 t/s, v2 4.6 t/s (same path). At 32 GiB with full arena: native 19.4 t/s, v1 5.5 t/s, v2 13.7 t/s. Details are in the report.
• Output matches native greedy decoding at 8 threads; at 16 threads it diverges after a few tokens (float reordering). See the report.