BoxyFit/Groot-65B-A6B-GyroM

Groot-65B-A6B — Gyro-M (TQ2_T)

120 tokens/s on a single RTX 3090. 20.2 GB total VRAM. Runs on a 12 GB card. Our own re-encode of Groot-65B-A6B's routed experts to trellis TQ2_T (2.25 bpw), encoded from the official FP8 teacher. First publisher of this combination (2026-11-03).

The number that matters is not the download size

The Groot team's own card told us a 72 GB Blackwell "can't fit" this model. That compares the download to your VRAM — they are different numbers. Here is the actual breakdown:

Component	Size	Lives in
Routed expert pool (TQ2_T, 2.25 bpw)	16.0 GB	GPU (hot tier) + host RAM (cold store)
N-gram lookup table (Q5_0, verbatim from official GGUF)	35.2 GB	host RAM only — never VRAM
Dense weights, embeddings (verbatim from official Q4_K_M)	~1.5 GB	GPU
Total file	54.5 GB	disk

The GPU never sees the 35 GB lookup table. Fully-resident VRAM on a 24 GB card: 20.2 GB (15.3 GiB expert cache + ~5 GB dense/KV/context), with 4 GB to spare.

Fit ladder — every rung measured on our 3090 (capping the expert cache to emulate smaller cards)

Real-card confirmation: the same pack runs fully resident on a Tesla V100-32 (~$500 used): 90 t/s engine decode, 12 GB VRAM spare — measured 2026-11-03 on a CUDA 12 build (CUDA 13 dropped Volta). A $500 2017 datacenter card is a 90 t/s Groot server.

Your GPU	Expert hot tier	Decode t/s	Measured hit rate (coding bench)
24 GB (RTX 3090/4090/A5000)	full 15.3 GiB	68–106 fresh context, 120–124 warm KV	100%, fully resident, 4 GB VRAM spare
16 GB (V100-16, 2080 SUPER class)	10.1 GiB (65% of pool)	80–95	89–98%
12 GB (3060-12, P40 class)	6.0 GiB (39%)	62–95	80–87%, PCIe traffic still ~0.1%
8 GB	~2.5 GiB	not yet measured	expect noticeable degradation

There is no speed cliff as the hot tier shrinks, because a cold expert fetch is only ~1.3 MB over PCIe — bandwidth, not residency, is the limit. For workload honesty: on a broad 1.25M-token corpus (IsValorum's independent Groot coverage data, 100% of experts observed), routing is flatter than on focused workloads: a 65% hot tier covers ~84% of activations, 39% covers ~63%. Speeds above already include that streaming cost.

System RAM: 32 GB works. The 35.2 GB lookup table does not need to be resident — measured, after real use only 5.7 GB of it (16%) is ever touched; evicting the whole table from page cache before every request cost zero measurable speed. The cold part lives on NVMe and is read sparsely. A fast NVMe is the real requirement; 16 GB RAM is the practical floor, 32 GB is comfortable. (The 128 GB in some reports is our bench box, not a requirement.)

Same size, different engine — measured contrast

We also benchmarked mradermacher's static Q2_K GGUF (56.3 GB, ~2.6 bpw — nearly the same file size as this pack) on the same RTX 3090 with stock llama.cpp (b11364, --cpu-moe + lookup table pinned to host RAM, because the 21 GB pool + KV does not fit 24 GB):

	this pack (TQ2_T, Strata)	Q2_K (llama.cpp)
decode	120 t/s	3.8–4.5 t/s
prefill	800 t/s	22.6 t/s
fast QA gate	10/10, direct answers	5/10 at short budget, 9/10 given 400 tokens — right answers buried under rambling

llama.cpp has no routing-aware expert cache: the pool is either fully resident (doesn't fit) or fully streamed over PCIe (~4 t/s). That gap is the engine, not the bits. For a llama.cpp-native path of this model, mradermacher's repo is the right link — this pack needs the Strata engine above.

Is it still smart?

Quant	Size class	HumanEval pass@1 (0-shot, temp 0, thinking)	MBPP pass@1	Decode on 1×RTX 3090
This pack — Gyro-M TQ2_T (2.25 bpw)	54.5 GB	95.1% (156/164)	93.4% (467/500)	120 t/s
Same pack, one Tesla V100-32 (~$500 used)	54.5 GB	same weights	same weights	90 t/s (fully resident, 12 GB VRAM spare)
Official Q4_K_M (4.8 bpw)	78.8 GB	98.2% (161/164)	not run	68–80 t/s
Community Q2_K, llama.cpp (2.56 bpw)	56.3 GB	fast-gate 5/10 short-budget, 9/10 with room	—	3.8–4.5 t/s

Same model, same harness (boxy-fastcoder-v1, official prompts + asserts, 2026-11-03). The requant at less than half the expert precision costs ~3 points on the coder bench and keeps 120 t/s — while the same-size llama.cpp 2-bit quant rambles and crawls at 4 t/s because it has no routing-aware hot cache.

• Groot is a reasoning model. Answers arrive only after a thinking trace; stacks with short output budgets or code stop-words truncate it mid-thought and score zero. If your eval said "Groot is dumb", check whether it read reasoning_content and gave the model room to finish — that explains a large share of such reports.
• Honest fine print: Groot's real failure mode is occasional reasoning-loop degeneration (the trace collapses into a repetition spiral and never emits code). 6 of this pack's 8 HumanEval misses are that mode, counted, not hidden.
• Encode quality: 11.35 dB signal-to-error at 2.25 bpw on real expert tensors (healthy for this rate; our production ladder threshold).
• Everything except the routed experts is byte-identical to the official Q4_K_M release — the lookup table, dense weights, and embeddings are copied verbatim.
• Detailed long benches (standard suites, long-context) are running; results will be added to this card and to the boxy.fit leaderboard (row groot-gyro, harness-stamped).

Quickstart (Strata engine)

This GGUF uses the trellis TQ2_T type (id 144) from the Strata/agention kernel family — upstream llama.cpp cannot load it. The filename carries a Q2_0 tag purely so HF's hardware estimator can parse it: Q2_0 is 2.25 bpw, the exact same rate as TQ2_T, so the size estimates are accurate. The tensors themselves are TQ2_T.

# 1. build the serving pack (seconds)
python tools/iq_pack.py --gguf Groot-65B-A6B-GyroM-Q2_0.gguf --compat-bf16 --out pack-groot-gyro
# 2. serve (Strata >= 0.2.8 / engine 0.1.54; CUDA arch 86+)
strata --pack pack-groot-gyro --native Groot-65B-A6B-GyroM-Q2_0.gguf \
       --expert-cache auto --spec 4 --kv k8v4 --max-context 32768

The engine sizes the hot tier to your free VRAM automatically — the 16 GB and 12 GB rows above are what --expert-cache auto produces on those cards.

What this is, exactly

Groot-65B-A6B is a whole-expert-table pruned Flash-Next (256 experts instead of 512, same per-token active path, ~6B active). We halved the stored expert pool again by re-encoding it with trellis quantization: experts re-encoded from the official FP8 teacher; all other tensors copied byte-for-byte from the official Q4_K_M GGUF. The result: the full 65B expert pool fits resident on one consumer card, and the model runs faster than it does streamed at 4-bit.

Run it yourself

The harness behind every number on this card is open: harness/fastcoder.py (HumanEval) and fastcoder_mbpp.py (MBPP). pip install datasets, point it at any OpenAI-compatible endpoint (python fastcoder.py <port> <model>), and you get the same prompts, same asserts, same RESULT line. Ran it on a card we don't have? Submit the RESULT line at boxy.fit — accepted claims go on the leaderboard with your credit.

Attribution & license

• Base model: GrootTeam/Groot-65B-A6B-261023 — thank you to the Groot team.
• llama.cpp quants of this model: mradermacher. Independent expert-coverage/imatrix data: IsValorum.
• Architecture and weights lineage: Qwen/Qwen3.8-Flash-Next.
• License: qwen-community-1.0 (text). This requant is a derivative work distributed under the same terms.
• Quantization tooling: Strata engine trellis kernels (Niko1221/Strata). Benchmarks: boxy.fit.