APEX quant benchmarks — 42-task agentic coding suite
Results for the GGUF models I quantize and benchmark, measured on a real agentic coding loop rather than a single-completion prompt. Every row below is a full 42-task run across all four suites.
This is a report, not a model. The quants themselves live in the individual
Myric/*-APEX-GGUF repositories, and the table links each row to the actual model
tested where a public card exists (otherwise to the base/official card — see the note
under the table).
The benchmark
42 tasks in four suites (private benchmark harness):
| suite | tasks | what it is |
|---|---|---|
opencode_tasks |
25 | general algorithmic ladder: data structures, graphs, DP, streams |
opencode_tasks_hard |
5 | fix/extend an existing codebase |
opencode_tasks_frontier |
9 | the hard-coder frontier (regex, btree, SQL, CRDT, rate limiters, txns) |
opencode_tasks_tier4a |
3 | multi-file repair in place (SWE-bench-style), not a harder rung |
How a run works. The model is served locally through llama-server and driven by
OpenCode's real tool-use loop (bash / read / write / edit) against each task's PROMPT.md.
The code it writes is graded by running the task's own pytest test_solution.py fresh —
the model's self-report is never trusted. Scoring is partial credit: the fraction of
individual pytest items passed, accumulated per suite.
Fixed conditions. --parallel 1 (a shared KV pool with concurrent OpenCode requests is
not comparable); compaction off; context auto-sized to the largest that fits up to
262,144; per-family output cap above worst-case reasoning; per-task wall 3,600 s; sampling at
the model card's recommended temperature/top-p.
Boxes. GB10 = NVIDIA GB10 (DGX Spark-class), ~119 GiB unified memory. Thor = a 2013 dual-Xeon workstation with 128 GB DDR3 and a 16 GB RTX 4060. The box is noted per row where it matters; wall-clock is only comparable within a box.
A perfect score is the entry bar, not a finding. These suites are designed to saturate for a competent coder. A full 42/42 says the arm qualified for comparison; it is not evidence that anything had no effect. The measurement lives in which suite a model falls off at, and — once pass/fail saturates — in output tokens at equal score. Code quality is not captured by this harness.
Noise. Seven of the nine frontier tasks can flip between identical repeats, and within-arm token spread runs to double digits. The table shows the best run per row and the Reps column gives the number of runs and the total range. A single-run row is a data point, not a noise floor. Tokens compare within a model, never across models of different scale.
Time is part of the measurement
A full 42-task run is typically ~2 hours on the GB10. Across the 74 full runs here the median is 1.8 h, p90 4.9 h, and the slowest outlier is 14.5 h (Spark-X2.5-4B BF16, 10 voids — each timeout can burn the full 3,600 s per-task wall). A 12-hour outer budget is therefore a real bound rather than a hypothetical: most runs never approach it, but a slow or verbose model that keeps timing out can exhaust it. A model that cannot finish inside that budget is, for this box, too slow to be useful — the benchmark is meant to answer the deployment question, not the abstract-capability question.
So a missing perfect score is not always a missing capability:
- A task that hits the wall or the output cap is voided, not marked wrong. The model may
have been on the right track and simply not finished. That is why
Valid = 42 − voidis the column to read, and why a row like Spark-X2.6-4B is 39/39 valid despite a 39/42 raw total. - But a void is also not evidence of capability. A model that voids the frontier tasks has not demonstrated that it can do them under the budget; a 100% valid ratio on a shrunken denominator is not the same as a clean 42/42. The clean rows are the stronger result.
- A smarter-but-slower model could move up on faster hardware or with a longer budget. The table measures capability under this GB10's time budget, not capability in the abstract. For a local deployment, the budget is part of the requirement.
This is why the frontier column and the Void count sit next to the score: they separate "did
not finish" from "got it wrong", and neither is silently folded into the other.
Full 42-task results
G = general (25), H = hard (5), F = frontier (9), T4A = tier4a (3). Per-suite and
total columns are the best run (most valid tasks passed); Void is that run's harness-limit
count and Reps gives run count and the total-passed range.
A void is not a failure. It is a timeout, output-cap truncation, or no-completion that the
harness refuses to score (n_valid = 42 − void). The Valid column is therefore
passed / (42 − void), and it is the number to read. The 42-slot denominator is kept only so
the void count is visible. This is not academic: Spark-X2.6-4B is 39/39 valid (3 void),
not 39/42; Spark-X2.5 BF16 is 32/32 with 10 void; MiniCPM5-2B is 35/35 with 7 void. Reporting
those as 39/42, 32/42, or 35/42 would count harness limits as model failures.
| Model | Quant | G (25) | H (5) | F (9) | T4A (3) | Valid | Void | Reps |
|---|---|---|---|---|---|---|---|---|
| Muse-Glimmer-30B | UD-Q4_K_XL | 25/25 | 5/5 | 9/9 | 3/3 | 42/42 | 0 | 1 |
| K2-Horizon-MoVA-36B-A4B | APEX i-quality v2 | 25/25 | 5/5 | 9/9 | 3/3 | 42/42 | 0 | 1 |
| Kolibri-1 78B-A3.5B | APEX 2.4 bpw | 25/25 | 5/5 | 9/9 | 3/3 | 42/42 | 0 | 1 |
| Kolibri-1 78B-A3.5B | Q4_K uniform | 25/25 | 5/5 | 9/9 | 3/3 | 42/42 | 0 | 2 (42) |
| Laguna-XS-2.1 | APEX i-quality v2 | 25/25 | 5/5 | 9/9 | 3/3 | 42/42 | 0 | 2 (42) |
| Laguna-XS-2.1 | APEX dynamic v2 | 25/25 | 5/5 | 8/9 | 3/3 | 41/42 | 0 | 2 (41) |
| Laguna-S-2.1 | APEX i-quality v2 | 25/25 | 5/5 | 7/7 | 3/3 | 40/40 | 2 | 1 |
| Nemotron-3-Super-120B-A12B | APEX i-quality | 25/25 | 5/5 | 9/9 | 3/3 | 42/42 | 0 | 1 |
| Ornith-1.5-122B | APEX i-quality | 25/25 | 5/5 | 9/9 | 3/3 | 42/42 | 0 | 2 (42) |
| Ornith-1.5-122B | APEX i-compact | 25/25 | 5/5 | 9/9 | 3/3 | 42/42 | 0 | 2 (42) |
| Ternary-Bonsai-2-27B | PQ2_0 (2.13 bpw) | 25/25 | 5/5 | 8/8 | 3/3 | 41/41 | 1 | 1 |
| Gemma-4-12B-it QAT | Q4_0 (grid) | 24/25 | 5/5 | 9/9 | 3/3 | 41/42 | 0 | 1 |
| Gemma-4-12B-it QAT | Q4_0 (Google) | 24/25 | 5/5 | 8/8 | 3/3 | 40/41 | 1 | 2 (39–40) |
| Mellum2.1-12B-A2.5B-Thinking | Q8_0 | 25/25 | 5/5 | 8/8 | 3/3 | 41/41 | 1 | 2 (40–41) |
| Mellum2.1-12B-A2.5B-Thinking | APEX 4.9 bpw | 25/25 | 5/5 | 8/9 | 3/3 | 41/42 | 0 | 2 (41) |
| Mellum2.1-12B-A2.5B-Thinking | APEX 4.3 bpw | 25/25 | 5/5 | 8/9 | 3/3 | 41/42 | 0 | 2 (40–41) |
| Mellum2.1-12B-A2.5B-Thinking | APEX 3.7 bpw | 25/25 | 5/5 | 9/9 | 3/3 | 42/42 | 0 | 2 (41–42) |
| Mellum2.1-12B-A2.5B-Thinking | APEX 3.1 bpw | 25/25 | 5/5 | 7/9 | 3/3 | 40/42 | 0 | 2 (36–40) |
| Mellum2.1-12B-A2.5B-Thinking | Q4_K_M stock | 25/25 | 5/5 | 8/9 | 3/3 | 41/42 | 0 | 2 (40–41) |
| Mellum2.1-12B-A2.5B-Thinking | Q6_K_L bartowski | 25/25 | 5/5 | 8/9 | 3/3 | 41/42 | 0 | 2 (41) |
| GPT-OSS-20B | MXFP4 (Thor) | 25/25 | 5/5 | 8/9 | 3/3 | 41/42 | 0 | 2 (40–41) |
| Nemotron-3.5-Lightning-30B-A3B | APEX compact (Thor) | 25/25 | 5/5 | 9/9 | 3/3 | 42/42 | 0 | 2 (40–42) |
| Gemma-4-26B-A4B | Q4_0 QAT (Thor) | 25/25 | 5/5 | 9/9 | 3/3 | 42/42 | 0 | 2 (41–42) |
| Kolibri-1 78B-A3.5B | APEX i-quality high (Thor) | 25/25 | 5/5 | 9/9 | 3/3 | 42/42 | 0 | 1 |
| Kolibri-1 78B-A3.5B | APEX i-quality low (Thor) | 25/25 | 5/5 | 9/9 | 3/3 | 42/42 | 0 | 1 |
| LFM2.5-24B-A2B | QAD Q4_0 | 25/25 | 5/5 | 7/9 | 3/3 | 40/42 | 0 | 4 (37–40) |
| LFM2.5-24B-A2B | Q8_0 | 25/25 | 3/3 | 8/8 | 3/3 | 39/39 | 3 | 1 |
| LFM2.5-24B-A2B | IQ3_XXS (BF16 src) | 25/25 | 4/4 | 7/9 | 3/3 | 39/41 | 1 | 2 (36–39) |
| Nemotron-3.5-Bolt-74B-A9B | 24 GB quant | 24/24 | 5/5 | 7/7 | 3/3 | 39/39 | 3 | 1 |
| Spark-X2.6-4B | Q6_K | 25/25 | 4/4 | 7/7 | 3/3 | 39/39 | 3 | 1 |
| Spark-X2.5-4B | BF16 | 22/22 | 3/3 | 4/4 | 3/3 | 32/32 | 10 | 1 |
| Spark-X2.5-4B | Q6_K | 22/22 | 3/3 | 2/2 | 3/3 | 30/30 | 12 | 1 |
| MiniCPM5-2B | Q8_0 | 23/23 | 3/3 | 6/6 | 3/3 | 35/35 | 7 | 1 |
Every row links to the public card for the exact quant where one exists; where the tested GGUF is local-only (the Gemma grid variant, Ornith's APEX tiers before publication, LFM's BF16-source IQ3_XXS), the link goes to the base/official card and the Quant column names the variant.
What the fall-off says
With voids excluded, the picture is cleaner — and different from the raw denominators:
- Clean 42/42, no voids: Muse-Glimmer-30B, K2-Horizon v2, Kolibri-1 (APEX and uniform), Laguna-XS i-quality v2, Nemotron-3-Super-120B, Ornith-1.5-122B (i-quality and i-compact). Their comparison is tokens-at-equal-score, not correctness.
- Perfect on every valid task, via voids: Spark-X2.6-4B (39/39, 3 void), Spark-X2.5 at BF16 and Q6_K (32/32 and 30/30; 10 and 12 void), MiniCPM5-2B (35/35, 7 void), LFM2.5 Q8_0 (39/39, 3 void), Nemotron-3.5-Bolt-74B (39/39, 3 void), Laguna-S (40/40, 2 void), and Bonsai 2 27B (41/41, 1 void). The voids are the output cap or the wall on the longest tasks; the write-ups note no degeneration. These are not capability losses. Bonsai's one void is the "smart but too slow" case: 56.8k output tokens and the full 3600 s wall, still unfinished — not a wrong answer.
- Real frontier misses (1–2 valid tasks): Laguna-XS dynamic v2 41/42, Mellum2.1 APEX 4.9 / 4.3 / stock Q4_K_M / bartowski Q6_K_L all 41/42, Mellum2.1 APEX 3.1 40/42, LFM2.5 QAD Q4_0 40/42, LFM2.5 IQ3_XXS 39/41. This is the scale doing its job: the frontier tier is where the quant or the smaller model gives up first.
- General-suite miss: Gemma-4-12B QAT grid 41/42 (one general task). The Google QAT arm is 40/41 on its best-passed rep (one general miss, one void); its second rep is 39/39 with 3 void. The general-suite miss is the one real failure; the void is a harness limit.
- Mellum2.1 APEX 3.7 bpw reaches 42/42 on its best rep (41/42 on the other), while 3.1 bpw falls to 40/42 — the aggressive tier is where the quant starts to cost correctness, and it shows up in the frontier column.
Plots
Measured wall-clock from the full 42-task runs. Time plots use GB10 runs only — Thor wall-clock is not comparable to the GB10. Each point is the best valid run for that model/quant. Parameter counts are nominal from the model cards/configs; Ornith-1.5-122B's active count is derived from its 256-expert top-8 config (~12 B).
Wall clock vs active parameters. Active count alone explains little: the 4 B Spark pair spans 2.5–14.5 h while the 12 B-active 120–124 B MoEs finish in 1.8–3.4 h.
Wall clock vs total parameters. Total size is not the driver either: a 124 B model can finish in 1.75 h while a 4 B model takes 14.5 h.
Wall clock vs output tokens. The clearest single driver: the runs that spend more output tokens take longer. The Spark outliers are the high-token runs, not the large models.
Verbosity vs active parameters. Output tokens per valid task; circles are GB10, squares Thor. Larger active models are not uniformly more verbose, and the 4 B Spark BF16 run is the outlier at ~18 k tokens/task.
Performance per byte. Capability vs quant file size (left; top-left is best) and end-to-end
output tokens/s vs size (right; GB10 only — throughput is box-specific). The ternary Bonsai 27B
PQ2_0 is the efficiency outlier: 41/41 valid at 7.21 GB, with one void
(btree_insert_delete — 3600 s / 32k cap). Full capability on every valid task, at the cost of
raw decode speed from the custom ternary kernels.
Also measured (subset runs)
Not comparable to the 42-task table above — these ran one suite, or fewer tasks. Kept here so the frontier signal is not lost.
| Model | Quant | Suite | Score | Note |
|---|---|---|---|---|
| Qwen3.8-Flash-Next | APEX (several arms) | frontier | 9/9 | one arm 8/9 |
| K2-Horizon-MoVA-36B-A4B | APEX compact / mini | frontier | 9/9 | both tiers |
| K2-Horizon-MoVA-36B-A4B | APEX i-quality | frontier | 8/9 GB10, 9/9 Thor | box difference |
| Ornith-1.5-35B-A3B | APEX i-quality | frontier | 9/9 | — |
| Nemotron-3.5-Lightning-30B-A3B | APEX i-quality | frontier | 7/9 | — |
| Spark-X2.5-4B | Q5 (1M-context) | general | 10/13 | long-context variant |
| Spark-X2.5-4B | Q4 | hard | 3/5 | — |
Quantization note
APEX tiers are per-tensor allocations, not fixed recipes: the edge/near/middle bands, the
shared-expert and latent-bottleneck protection, and the F32 state/norm/router pins all change
per architecture. i-quality is a shelf label for the tier I reach for by default, not a
claim of identity with another project's recipe. Sizes and bpw live in each model repo.
Files
| path | contents |
|---|---|
results.json |
every full-42 and subset run, per-suite passed/valid/partial, model path, box, void/wall |
results_full42.csv |
flat table of the 74 full-42 runs, one row per run |
plots/ |
wall-clock vs parameters/tokens and verbosity plots |
Caveats
- Pass/fail is a gate. This harness saturates by design; a perfect score is not a ranking.
- Code quality is unmeasured. Tests-passed and tokens are what is recorded; clarity, structure, and whether the model understood the problem are not.
- Tokens are within-model. Cross-model token comparisons confound capability with verbosity and are not reported.
- Single runs are noisy. Where only one run exists, treat the score as a data point.