christopher-kapic/MiMo-V2.5-ROCmFP4-GGUF

MiMo-V2.5 — ROCmFP4 GGUF (Strix Halo / gfx1151)

ROCmFP4 quantizations of MiMo-V2.5 (310B total / 15B active MoE), built for AMD Strix Halo (Ryzen AI Max+ 395, gfx1151) with the ROCmFPX fork of llama.cpp.

These are made for people running MiMo-V2.5 across two 128 GB Strix Halo boxes over a USB4/Thunderbolt link using llama.cpp's RPC layer split. The model does not fit on one node at this precision.

Note on quality: these are not imatrix-calibrated. Unsloth's UD-Q4_K_XL is, and it is tensor-aware on top of that. The ROCmFP4 variants here win on speed and size, but if output quality at the margins matters more to you than tok/s, benchmark both before switching.

⚠️ Requirements — stock llama.cpp will not load these

Q4_0_ROCMFP4 and Q4_0_ROCMFP4_FAST are quantization types defined by the ROCmFPX fork. Upstream llama.cpp, Ollama, LM Studio and every downstream that vendors mainline ggml cannot read these files — you will get an unknown-ggml-type error, not a slow model. You must build the fork.

They also target gfx1151 specifically (Ryzen AI Max+ 395 / Strix Halo). The formats are built around that hardware's dequant path; on other GPUs, expect either a build failure or no benefit.

Build

git clone https://github.com/charlie12345/ROCmFPX.git
cd ROCmFPX
git checkout 3edc3d31ee5ebcea47fd7e0f42c89767bb4245db   # the commit these were built and tested with

cmake -S . -B build \
    -DCMAKE_BUILD_TYPE=Release \
    -DGGML_HIP=ON \
    -DGGML_RPC=ON \
    -DGGML_HIP_FORCE_MMQ=ON \
    -DGGML_HIP_ROCWMMA_FATTN=OFF \
    -DGGML_VULKAN=OFF -DGGML_CUDA=OFF \
    -DCMAKE_HIP_ARCHITECTURES=gfx1151 \
    -DGPU_TARGETS=gfx1151 \
    -DLLAMA_BUILD_SERVER=ON \
    -DLLAMA_BUILD_WEBUI=OFF -DLLAMA_USE_PREBUILT_WEBUI=OFF \
    -DLLAMA_BUILD_TESTS=OFF -DGGML_BUILD_TESTS=OFF

cmake --build build -j "$(nproc)" --target \
    llama-cli llama-server llama-bench llama-quantize rpc-server

-DGGML_RPC=ON is what gives you rpc-server and the RPC0 device — required for the two-node split. Built against ROCm 6.4.

Use the ROCm backend, not Vulkan. RADV imposes a per-buffer allocation ceiling that a model this size runs straight into; -dev Vulkan0 will OOM where ROCm0 works fine.

Making the memory available

A single Strix Halo box defaults to a GPU carve-out far below what these need. Either set GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 in the environment, or raise the GTT limit on the kernel command line (what we run):

amdgpu.gttsize=126976 ttm.pages_limit=32505856 ttm.page_pool_size=32505856

With the cmdline set, UNIFIED_MEMORY is no longer needed. We measured no throughput difference between the two approaches — the cmdline is just less fragile.

⚡ Force the GPU power level — worth 14% decode

This is the single highest-value host setting, and it is easy to miss.

echo high | sudo tee /sys/class/drm/card*/device/power_dpm_force_performance_level

power_dpm_force_performance_level	pp512	tg128
auto (default)	322.87	20.94
high	329.43	23.87

+14% decode, +2% prefill, from one sysfs write. The DPM governor never ramps to peak on MoE decode — the workload is bursty and low-occupancy, so auto reads it as near-idle and leaves the GPU and fabric below spec. Effective memory bandwidth goes from 167 to 190.7 GB/s (65% -> 75% of the 256 GB/s theoretical). DRAM is at its rated 8000 MT/s either way; this is purely a clock-governor effect.

The gain is specific to single-stream decode. Under concurrency the GPU is already loaded enough that auto ramps by itself — aggregate throughput at C8 is unchanged.

Transparent hugepages were tested alongside this and make no difference (21.24 with THP alone vs 20.94 baseline), despite the 153 GiB working set. Not worth the system-wide side effects.

It resets on reboot. To persist:

# /etc/systemd/system/amdgpu-perf-high.service
[Unit]
Description=Force amdgpu DPM to high
After=multi-user.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/sh -c 'for c in /sys/class/drm/card*/device/power_dpm_force_performance_level; do echo high > "$c"; done'
ExecStop=/bin/sh -c 'for c in /sys/class/drm/card*/device/power_dpm_force_performance_level; do echo auto > "$c"; done'

[Install]
WantedBy=multi-user.target

Apply it on every node — the RPC peer's clocks matter just as much as the head node's.

Variants

variant	effective bpw	size	shards	folder
Q4_0_ROCMFP4_FAST ← recommended	4.26	153.47 GiB (164,795,562,720 B)	4	ROCmFP4-FAST/
Q4_0_ROCMFP4	5.24	189.02 GiB (202,969,561,824 B)	5	ROCmFP4/

For reference, unsloth/MiMo-V2.5-GGUF UD-Q4_K_XL is 178.44 GiB at 4.95 bpw — ROCmFP4_FAST is 14% smaller.

ROCmFP4_FAST dominates ROCmFP4 on every axis we measured — smaller, faster prefill, faster decode. Unless you specifically want the higher-precision tensors, take FAST. ROCmFP4 is published for completeness.

A note on the bpw labels. llama-quantize advertises Q4_0_ROCMFP4 as "4.50 bpw", but measured against the 309.77 B parameter count the real figure is 5.24 bpw — the recipe promotes several tensor classes (e.g. ffn_gate → q5_K) rather than quantizing everything to ROCmFP4. Q4_0_ROCMFP4_FAST measures 4.26 bpw against its 4.25 label, so that one is honest. This is why the "non-fast" build ends up larger than UD-Q4_K_XL rather than smaller.

Both were quantized from the BF16 GGUF (unsloth/MiMo-V2.5-GGUF, 14 shards, 619,638,702,336 bytes, verified byte-exact) with:

llama-quantize MiMo-V2.5-BF16-00001-of-00014.gguf \
    MiMo-V2.5-ROCmFP4-FAST.gguf Q4_0_ROCMFP4_FAST 16

Measured performance

Test setup

Two Beelink GTR 9 Pro (Ryzen AI Max+ 395, gfx1151, 128 GB unified) linked by a single USB4 cable, layer-split with llama.cpp RPC. All numbers are single-stream.

engine	ROCmFPX (llama.cpp fork), ROCm 6.4, HIP backend
OS / kernel	Ubuntu 24.04, mainline 6.18.6
topology	node2 = head, node1 = rpc-server over Thunderbolt (192.168.2.1:50052)
GPU carve-out	126976 MiB via amdgpu.gttsize / ttm.pages_limit kernel cmdline
transport	TCP over thunderbolt0 (RDMA measured, no difference)

Benchmark command:

llama-bench -m <model.gguf> \
    -rpc 192.168.2.1:50052 -dev ROCm0/RPC0 \
    -ngl 999 -fa 1 -mmp 0 -r 1 -p 512 -n 128

-mmp 0 (no mmap) is required — with mmap the working set thrashes against the 128 GB of RAM and never converges. Note llama-bench wants -dev entries separated by /, while llama-cli wants ,.

Single-stream results

All rows below are measured with power_dpm_force_performance_level=high on both nodes (see Requirements) — without it every number drops 9–14%.

variant	size	prefill (pp512)	decode (tg128)
Q4_0_ROCMFP4_FAST	153.47 GiB	328.73 t/s	23.80 t/s
(reference) UD-Q4_K_XL	178.44 GiB	339.31 t/s	17.16 t/s
Q4_0_ROCMFP4 (at dpm=auto)	189.02 GiB	248.10 t/s	16.80 t/s

ROCmFP4_FAST is +38.7% decode over UD-Q4_K_XL for −3.1% prefill, while being 14% smaller. The decode gain far exceeds what the size reduction alone predicts (4.26 vs 4.95 bpw) — the single-scale layout also dequantizes more cheaply on gfx1151.

Both were re-measured at dpm=high so the comparison is like-for-like. Worth noting UD-Q4_K_XL gains only ~9% from that tuning where ROCmFP4_FAST gains 14%, consistent with the FAST dequant path being more clock-sensitive. At the old dpm=auto default the gap read as +33%.

ROCmFP4 (non-fast) has not been re-measured at dpm=high; its row is from dpm=auto and is not comparable to the two above. It was the weakest of the three on throughput at equal settings and is published only for completeness.

Reproducibility: ROCmFP4_FAST at dpm=high was measured three times — 329.43/23.87, 329.14/23.89, 328.73/23.80 — a 0.2% spread on pp512 and 0.4% on tg128.

Decode vs. context depth

Q4_0_ROCMFP4_FAST, with power_dpm_force_performance_level=high:

depth	pp512	tg128
0	328.73	23.80
8192	286.70	23.00
32768	214.53	21.72

Decode is remarkably flat with context — a consequence of the 9-full/39-SWA attention split, where only 9 layers grow with depth. Prefill decays normally.

For reference, the same curve at the default dpm=auto was 315.5/21.07, 267.1/20.68 and 197.7/19.53 — the tuning is worth ~11–13% of decode at every depth.

Aggregate throughput under concurrency

Total tokens/s across all streams, 16 distinct prompts so that slots cannot share a prefix-cache hit and inflate the result (--parallel 8, -c 32768).

variant	C4	C6	C8
Q4_0_ROCMFP4_FAST (dpm=high)	33.20 t/s	34.51 t/s	33.57 t/s
Q4_0_ROCMFP4_FAST (dpm=auto)	31.93 t/s	32.61 t/s	33.92 t/s
Q4_0_ROCMFP4 (dpm=auto)	24.96 t/s	26.37 t/s	28.32 t/s
(reference) UD-Q4_K_XL (dpm=auto)	25.81 t/s	29.19 t/s	31.72 t/s

Aggregate throughput plateaus around 33–35 t/s and the dpm=high tuning barely helps here — under concurrency the GPU is already busy enough that the governor ramps on its own. That is the mirror image of the single-stream case, where forcing high is worth 14%.

ROCmFP4_FAST is fastest at every concurrency level, but its margin over UD-Q4_K_XL shrinks as concurrency rises — the FAST layout's advantage is in memory bandwidth and dequant cost, which dominate single-stream decode; as batch size grows the workload shifts toward expert scatter and compute, where the quants converge.

MiMo-V2.5 is a 256-expert top-8 MoE, the least favourable case for batch amortization: the number of distinct experts touched at batch B grows as 256·(1−(1−8/256)^B), so more expert weights must be read as concurrency rises instead of being amortized across the batch. This is why aggregate throughput scales so weakly — ROCmFP4_FAST gains only 6% going from 4 streams to 8, and per-stream latency roughly halves over that range (7.98 → 4.24 t/s).

Usage

Serve across two nodes (run rpc-server on the second box first):

# node1
rpc-server -H 0.0.0.0 -p 50052

# node2
llama-server -m ROCmFP4-FAST/MiMo-V2.5-ROCmFP4-FAST-00001-of-00004.gguf \
    --rpc 192.168.2.1:50052 -dev ROCm0,RPC0 \
    -ngl 999 -fa 1 --no-mmap -c 32768 --parallel 8 --host 0.0.0.0

Point llama.cpp at the first shard; it pulls in the rest automatically.

Mind the flag spelling — llama-server takes --rpc and comma-separated -dev, while llama-bench takes -rpc and slash-separated -dev (ROCm0/RPC0). Passing the wrong one makes the server exit immediately with error: invalid argument.

Things worth knowing

• Keep KV cache at f16 for speed. Measured on this exact model and split:

  -ctk / -ctv	tg128 @d0	tg128 @32k
  f16 / f16	20.71	19.44
  q8_0 / q8_0	20.26	17.13
  q8_0 / q4_0	20.17	17.24
  q4_0 / q4_0	20.04	17.22

  The penalty is ~2% at zero depth but 13% at 32k — ROCm dequant costs more than the bandwidth it saves, and the gap widens as the cache fills. Note the three quantized configs are indistinguishable: the cost comes from quantizing at all, not from how aggressively.

  But quantizing buys context. f16 KV is 22.5 KiB/token here; q4_0/q4_0 is ~6.6 KiB — about 3.4× the KV pool for that 13%. If maximum context matters more to you than decode speed, -ctk q4_0 -ctv q4_0 is the trade.

• Prefer the USB4/Thunderbolt link over ethernet for the RPC hop. Against a switched 1 GbE path we measured −5% decode and −9.5% prefill. The causes differ: decode is latency-bound (llama.cpp RPC does ~3–4 round trips per token, so hop latency multiplies), while prefill is bandwidth-bound (4 MB of activations per 512-token chunk). A switched 10 GbE link fixes the prefill half but not the decode half; a direct point-to-point cable fixes both.

• Speculative decoding does not pay off on this model. Measured on this exact build and split, with --temp 0 --repeat-penalty 1.0:

  workload	no speculation	DFlash n=1	DFlash n=2	DFlash n=4
  structured JSON	21.00	14.87	13.80	10.02
  Rust code	21.18	13.00	11.96	8.94
  narrative prose	21.21	12.19	10.72	8.16

  DFlash costs 30–60%, and gets worse the deeper you draft. The cause is acceptance. Mean accept length is 1.52 / 1.32 / 1.23 (JSON / code / prose), and per-position acceptance on JSON runs 0.500, 0.150, 0.011, 0.000 — the drafter is right about half the time on token 1 and essentially never by token 3.

  A DFlash step costs 2.15× a normal decode step here, so break-even needs accept length ≥ 2.15. Reference implementations on other engines report 3.78 on this same model and drafter, which would be ~1.76× — so the headroom is real, it is just not reachable from this engine. An independent DFlash draft GGUF for MiMo-V2.5-Pro on ik_llama.cpp reports the same shape (54.6–60.4% acceptance, 55.6–59.4 t/s drafted vs 59.9–60.8 undrafted — also a net loss).

  Measured with thinking off; enabling it changes JSON and prose by ~0 and costs code about 16%.

  MTP is unavailable (llama.cpp issue #23924 closed not_planned, though these GGUFs do carry the blk.48-50.nextn.* tensors), and no EAGLE3 drafter has been published for MiMo-V2.5.

• If you do experiment with DFlash over an RPC split, the target's LM head and token embeddings must be pinned to the local device or it aborts at load in ggml_backend_sched_backend_id_from_cur: -ot "output\.weight=ROCm0" -ot "token_embd\.weight=ROCm0" -devd ROCm0

• KV geometry: MiMo-V2.5 has 9 full-attention + 39 sliding-window layers, so only the 9 full layers scale with context — about 22.5 KiB/token.

License

Inherits the license of the base model, XiaomiMiMo/MiMo-V2.5. Quantization adds no additional restrictions.

Credits

• XiaomiMiMo — MiMo-V2.5
• unsloth — BF16 GGUF conversion used as the source
• ROCmFPX — the ROCmFP4 quant formats