Quacken-35B-A3B-FP8

The Rock8 - Got any weights? 💪🦆

Native fp8 E4M3 GGUF of Qwen3.6-35B-A3B - a Mixture-of-Experts GatedDeltaNet-hybrid (multimodal-capable) model - for AMD RDNA4 (gfx1201 - Radeon AI PRO R9700 / RX 9070 / 9070 XT / W-series), quantized with AMD Quark from the full-precision BF16 weights by The Rock8.

This is authentic Quark MoE fp8: the fused experts themselves are quantized to fp8, not just attention. The Rock8's llama.cpp fork runs it on RDNA4's native WMMA fp8 tensor cores (MUL_MAT_ID for the experts) - no dequant-to-f16 fallback.

What it is

• Format: fp8 E4M3 (F8E4M3), block-scaled, produced by AMD Quark from BF16.
• Architecture: MoE (A3B active) + GatedDeltaNet SSM-hybrid; has an MTP head (usable for self-speculative decode).
• Target: AMD RDNA4 / gfx1201; ~38.7 GB -> runs 2-GPU (tensor-split across two 32 GB R9700s) or one large card.
• Runtime: The Rock8 (llama.cpp fork with native RDNA4 fp8 + fp8-MoE kernels) on TheRock ROCm 7.13.
• File: qwen3.6-35b-a3b-quark-fp8-moe-authentic.gguf (38.7 GB).

Source model + license

• Source: Qwen3.6-35B-A3B (Qwen).
• License: Apache-2.0 (redistribution of this quantized derivative is permitted with attribution). This is a derivative work.

Validation (real gfx1201 hardware)

Metric	Value
Perplexity (wikitext, 20 chunks, n_ctx=512), 2-GPU	6.6270
Prefill pp512, 2-GPU	3197 t/s
Decode tg128 (single-stream), 2-GPU	70.2 t/s
Aggregate decode @ peak concurrency (npl=114)	~537 t/s (8.2×)

Validated via llama-perplexity on dual R9700 (gfx1201). This is an SSM-hybrid architecture: use llama-server / llama-perplexity, not llama-bench.

Multi-user serving — the MoE's real strength (vLLM replacement on RDNA4)

This is a sparse MoE (~3B active of 35B), which makes it the batch-prone member of the family: as concurrency rises, more tokens share the same expert weight reads (expert-union amortization), so aggregate throughput scales hard. Measured with llama-batched-bench on dual R9700:

Concurrent seqs (npl)	Aggregate decode (tok/s)	Scaling
1	65.8	1.00×
8	249.9	3.80×
32	355.3	5.40×
64	432.2	6.57×
110	531.5	8.08×
114	537.4	8.17× ← peak
118	535.4	8.14×

One dual-R9700 box serves ~110 concurrent users at ~537 tok/s aggregate. Decode climbs to npl≈114 (8.2× single-stream), then the MoE's routed-expert union saturates (all 256 experts pulled per step → no further weight-read amortization) and it knees — not a VRAM wall (npl=128 fits). Throughput oscillates ~±11% with npl mod 4 (a batch/ubatch-tiling alignment effect, reproducible across runs): npl ≡ 2 mod 4 is the favorable alignment (110/114/118 = 531–537 t/s) and ≡ 0 mod 4 is worst (112/116 = 451–490) — so pick a ≡2-mod-4 --parallel value. Same continuous-batching amortization that was vLLM's "server win," but native and with fp8 (vLLM silently dequantizes fp8 on gfx1201, so its edge evaporates here). Note: fp8 KV-cache (-ctk/-ctv f8e4m3) is incompatible with batched decode on this hybrid-SSM MoE (breaks B>1) — batching is f16-KV only. For MoE, reach for concurrency (this section); for single-user latency, MTP below.

Run it (2-GPU)

# -ngl 999 lets llama.cpp see and tensor-split across both R9700s
llama-server -m qwen3.6-35b-a3b-quark-fp8-moe-authentic.gguf -ngl 999 --host 0.0.0.0 --port 13305
# it has an MTP head -> self-speculative decode for faster single-stream latency:
llama-server -m qwen3.6-35b-a3b-quark-fp8-moe-authentic.gguf -ngl 999 --spec-type draft-mtp

# multi-user: continuous batching for concurrent serving (the MoE's strength)
# (f16-KV only — fp8-KV breaks batched decode on this hybrid-SSM MoE)
llama-server -m qwen3.6-35b-a3b-quark-fp8-moe-authentic.gguf -ngl 999 \
  --cont-batching --parallel 114   # peaks ~537 tok/s aggregate (npl=114, a ≡2-mod-4 value)

curl -s http://localhost:13305/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"What do you call a dried grape? Answer in one word."}],"max_tokens":16}'
# expect: raisin

Lemonade appliance (container)

podman run -d --rm --runtime crun --name lemonade \
  --device /dev/kfd --device /dev/dri \
  --group-add keep-groups --security-opt seccomp=unconfined \
  -v /path/to/quacken-35b:/models:ro \
  -e MODEL=/models/qwen3.6-35b-a3b-quark-fp8-moe-authentic.gguf -e MODEL_NAME=Quacken-35B-A3B-FP8 \
  -p 13305:13305 \
  ghcr.io/the-monk/the-rock8:rdna4-tr713 serve
# 35B needs both GPUs - do NOT pin HIP_VISIBLE_DEVICES to a single card

Container (same image on each registry; --runtime crun is required for GPU): ghcr.io/the-monk/the-rock8:rdna4-tr713 - docker.io/gorilla4x/the-rock8:rdna4-tr713 - quay.io/the-monk/the-rock8:rdna4-tr713 (images may not be pushed to every registry yet).

The Rock8 - RDNA4 fp8 (links)

• GitHub: The-Rock8 - kernels, patch series, appliance recipe, full feature doc.
• Collection: The Rock8 - RDNA4 fp8.
• Sibling models: Quacken-8B-FP8 - Quacken-R1-14B-FP8 - Quacken-27B-FP8 - Quacken-27B-NVFP4 (mixed FP4/FP8) - Quacken-Ornith-35B-FP8 - Bonsai-8B-Ternary-RDNA4 (async spec-decode).

Every artifact links to the others - land on any one, reach them all.