bakon3/Qwen3.6-35B-A3B-ROCMFP

Qwen3.6-35B-A3B — ROCmFPX sealed (AMD-calibrated)

Custom ROCmFPX GGUFs of Qwen/Qwen3.6-35B-A3B for AMD Radeon AI PRO R9700 (gfx1201) / ROCm HIP.

Built and bench’d on donherm: dual R9700 32 GB, ROCm 7.2.4, ROCmFPX HIP pin 45bcff5, Hermes agent lab workflow.

File	Type	Size
Qwen3.6-35B-A3B-Q4_0_ROCMFP4_COHERENT-imatrix.gguf	Q4_0_ROCMFP4_COHERENT + imatrix	~19 GB
Qwen3.6-35B-A3B-Q6_0_ROCMFPX_AGENT-imatrix.gguf	Q6_0_ROCMFPX_AGENT + imatrix	~31 GB
Qwen3.6-35B-A3B-imatrix.gguf	importance matrix	~184 MB

---

⚠️ Runtime requirement

Not stock GGUFs. Need ROCmFPX HIP llama.cpp (tested pin: 1337hero/ROCmFPX @ 45bcff5).

Will not load in stock llama.cpp / Ollama / LM Studio / Vulkan-only builds.

export LD_LIBRARY_PATH=/path/to/ROCmFPX/build/bin:/opt/rocm/lib:$LD_LIBRARY_PATH
# Prefer discrete GPUs: --device ROCm0 / ROCm1 (exclude iGPU if present)

---

Naming (format vs profile)

Token	Meaning
ROCMFP4 / ROCMFPX	AMD block format (needs ROCmFPX runtime)
COHERENT / AGENT	Recipe: heavier emb/output or agent/tool-biased tensor routing
-imatrix	Quantized with importance matrix

Examples:

• Q4_0_ROCMFP4_COHERENT = 4-bit ROCmFP4 + Q6_K embeddings + imatrix
• Q6_0_ROCMFPX_AGENT = 6-bit ROCmFPX + agent/coherent routing + imatrix

---

How they were made

Shared pipeline

1. Source: Unsloth BF16 GGUF of Qwen3.6-35B-A3B (~66 GB).
2. BF16 full load for imatrix is impossible on 2×32 GB (+ host RAM limit).
3. Built temporary Q8_0 from BF16 for activation collection only (deleted after).
4. llama-imatrix on dual R9700 (layer TP) over a mixed calib corpus:
   • code / systems prose
   • agent / tool JSON
   • math
   • light multilingual
     → *-imatrix.gguf (511 entries, 24 chunks; MoE experts may be partially covered).
5. Quantized from BF16 with --imatrix (not requant from Q4).

Quants

Output	Command ftype
Q4 sealed	Q4_0_ROCMFP4_COHERENT + --imatrix
High Q6	Q6_0_ROCMFPX_AGENT + --imatrix

Tooling: Automated quant, A/B harness, and HF packaging via lab automation (account bakon3).

---

Benchmarks (donherm lab)

Hardware & software (all benches unless noted)

Item	Value
GPUs	2× AMD Radeon AI PRO R9700 (gfx1201), 32 GB each
iGPU	present — excluded via --device ROCm0 / ROCm1
ROCm	7.2.4 @ /opt/rocm
Binary	ROCmFPX HIP build 45bcff5 (build 119)
Common flags	FA on, --no-mmap, q8_0 KV when server, batch 2048 / ubatch 512

A) Official llama-bench — Q4 sealed vs Unsloth UD-Q4

Topology: single GPU ROCm0, -ngl 99 -fa 1 -b 2048 -ub 512 -r 3
Control: Unsloth Qwen3.6-35B-A3B-UD-Q4_K_XL (~20.8 GiB)
Date: 2026-07-16

Test	Q4 COHERENT+imatrix	Unsloth UD-Q4	Δ
tg128	93.4	79.5	+17.5%
tg256	94.2	80.0	+17.7%
tg512	94.2	80.1	+17.5%
pp512	2708	2951	−8.2%
pp8192	2310	2568	−10.0%

B) Server A/B chat — Q4 sealed vs Unsloth (no ngram)

Topology: dual simultaneous — ROCm0=sealed Q4, ROCm1=Unsloth
Server: llama-server -c 32768 parallel 1, FA on, q8_0 KV, no ngram
Harness: OpenAI-compatible /v1/chat/completions (Hermes Python suite)
Date: 2026-07-16

Test	Sealed tg	Unsloth tg	Δ
decode_128	87.7	74.9	+17.2%
decode_256	88.6	74.6	+18.8%
decode_512	88.4	74.5	+18.6%
decode_1024	88.3	74.7	+18.3%
parallel both GPUs 256	88.5	74.9	+18.3%

Mean decode Δ: ~+18.2%

Quality smokes (same server suite)

Check	Sealed	Unsloth
17*19 → 323	✓	✓
123*45 → 5535	✓	✓
JSON object (name/language/functions/tests_pass)	✓	✓
iterative fib(n)	✓	✓
bat/ball → $0.05	✓	✓

C) Q6 AGENT dual-TP @ 256k (high-ctx path)

Topology: one model, layer TP -sm layer -ts 1,1 on ROCm0+ROCm1
Why TP: ~31 GB weights cannot run as two full copies on 2×32 GB
Server: -c 262144, FA on, q8_0 KV, ngram-mod 24/48/64, parallel 1, batch 2048/512
Date: 2026-07-17 · harness: same chat suite via Hermes

Test	avg tg	notes
decode_128	~107	ngram bimodal (cold ~67, peak ~140)
decode_256	~102	cold ~69 / peak ~128
decode_512	~97	cold ~69 / peak ~140
decode_1024	~73	less draft help
repetitive code 512	~226	peak ~385 with ngram
steady open decode	~68–70	without draft hits

Prefill (chat long-prompt path): ~166 → ~133 pp as prompt grows ~0.6k→20k tokens.

Quality: all pass (323, 5535, JSON, fib, $0.05).

---

Launch recipes

BIN=/path/to/ROCmFPX/build/bin/llama-server
export LD_LIBRARY_PATH=$(dirname "$BIN"):/opt/rocm/lib:$LD_LIBRARY_PATH
Q4=Qwen3.6-35B-A3B-Q4_0_ROCMFP4_COHERENT-imatrix.gguf
Q6=Qwen3.6-35B-A3B-Q6_0_ROCMFPX_AGENT-imatrix.gguf

Dual GPU — Q4 @ 256k native (one full copy per GPU)

"$BIN" -m "$Q4" --host 0.0.0.0 --port 8000 --device ROCm0 \
  -c 262144 -ngl 99 --parallel 3 --cont-batching --kv-unified \
  --flash-attn on --no-mmap --jinja \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --batch-size 2048 --ubatch-size 512 --cache-ram 4096 \
  --spec-type ngram-mod \
  --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64 \
  --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0 --verbosity 3 &

"$BIN" -m "$Q4" --host 0.0.0.0 --port 8001 --device ROCm1 \
  -c 262144 -ngl 99 --parallel 3 --cont-batching --kv-unified \
  --flash-attn on --no-mmap --jinja \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --batch-size 2048 --ubatch-size 512 --cache-ram 4096 \
  --spec-type ngram-mod \
  --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64 \
  --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0 --verbosity 3 &

Dual GPU — Q4 @ 512k YaRN (one full copy per GPU)

Add:

-c 524288 --rope-scaling yarn --rope-scale 2 --yarn-orig-ctx 262144 \
  --override-kv qwen35moe.context_length=int:524288 --context-shift

Keep batch 2048 / ubatch 512. Flash-attn may OOM on very long multi-request sessions — use --flash-attn off or lower parallel if needed.

Q6 AGENT — dual-TP 256k (single server)

"$BIN" -m "$Q6" --host 0.0.0.0 --port 8000 \
  --device ROCm0,ROCm1 -sm layer -ts 1,1 \
  -c 262144 -ngl 99 --parallel 1 \
  --flash-attn on --no-mmap --jinja \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --batch-size 2048 --ubatch-size 512 --kv-unified \
  --spec-type ngram-mod \
  --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64 \
  --verbosity 3

---

Download

# recommended Q4
hf download bakon3/Qwen3.6-35B-A3B-ROCMFP \
  Qwen3.6-35B-A3B-Q4_0_ROCMFP4_COHERENT-imatrix.gguf

# high Q6 agent
hf download bakon3/Qwen3.6-35B-A3B-ROCMFP \
  Qwen3.6-35B-A3B-Q6_0_ROCMFPX_AGENT-imatrix.gguf

# imatrix (repro)
hf download bakon3/Qwen3.6-35B-A3B-ROCMFP Qwen3.6-35B-A3B-imatrix.gguf

---

SHA256

See SHA256SUMS in this repo.

---

License / lineage

• Base: Apache-2.0 — Qwen/Qwen3.6-35B-A3B
• BF16 GGUF: unsloth/Qwen3.6-35B-A3B-GGUF
• Runtime: charlie12345/ROCmFPX / R9700 pin work
• Methodology cousin (dense 27B Q8): 1337Hero/Qwen3.6-27B-Q8_0-ROCMFPX-GGUF

Derivative research quants — not official Qwen/Unsloth releases.

Changelog

• 2026-07-17: Add Q6_0_ROCMFPX_AGENT-imatrix; recommended dual-GPU quant = Q4 COHERENT; full bench methodology; remove straight no-imatrix Q4.
• 2026-07-16: Initial sealed Q4 COHERENT+imatrix + A/B vs Unsloth.